Hands-on Apache Beam, Building Data Pipelines in Python

Introduction

A distributed or direct data pipeline can be constructed using Apache Beam, an open-source SDK, based on batch or stream-based integrations. For every pipeline, you can add different transformations. Nevertheless, Beam's true strength lies in its lack of reliance on any one compute engine, making it independent of platforms. In order to compute your transformation, you specify the "runner" to employ. You can select a Spark engine or Cloud Dataflow, for example, instead of the default use of your local computing resource.

By utilizing Beam's Python SDK, developers can construct complex data pipelines with simplicity and scalability. Apache Beam abstracts the execution layer, allowing users to run their pipelines on multiple execution engines like Google Cloud Dataflow, Apache Flink, and Apache Spark without altering the pipeline code. Key concepts include PCollections (data), PTransforms (operations), and I/O connectors for various data sources and sinks. The flexibility of Apache Beam makes it a powerful tool for handling large-scale data processing tasks, offering real-time and batch processing capabilities, thus making it essential for modern data engineering and analysis.

Installation

Apache Beam (2.8.1) is only compatible with Python 2.7 as of the writing of this article; a Python 3 version should be forthcoming shortly. Beam is known to crash if you have Python Snappy installed, however this will be fixed in Beam 2.9.

Basic Pipeline

A basic Apache Beam pipeline in Python involves three main steps: reading data, transforming it, and writing the results. You start by defining a `Pipeline` object, followed by creating a `PCollection` to hold the input data. Next, apply `PTransforms` to process the data, such as filtering, mapping, or aggregating. Finally, write the transformed data to an output sink. Apache Beam's flexibility allows running the same pipeline on different backends like Google Cloud Dataflow or Apache Flink, making it a versatile tool for batch and stream processing. This approach simplifies complex data workflows with scalable, efficient execution.

Example

 
import apache_beam as beam

# Define a function to transform the data
def transform_data(element):
    return element.upper()

with beam.Pipeline() as pipeline:
    # Read data from a text file
    input_data = pipeline | 'Read' >> beam.io.ReadFromText('input.txt')
    
    transformed_data = input_data | 'Transform' >> beam.Map(transform_data)
    
    
    transformed_data | 'Write' >> beam.io.WriteToText('output.txt')

result = pipeline.run()
result.wait_until_finish()

with open('output.txt-00000-of-00001', 'r') as f:
    output = f.read()

print(output)   

Output:

 
HELLO WORLD
APACHE BEAM
DATA PROCESSING
PIPELINE EXAMPLE

Explanation

This sample demonstrates a basic data transformation Apache Beam pipeline. If it isn't already installed, it starts by installing the Apache Beam library. Subsequently, a pipeline is defined, which carries out three primary tasks: receiving text data from `input.txt`, transforming each line to be capitalized, and publishing the changed lines to `output.txt{. The pipeline operates asynchronously and waits for completion when it is configured. Following processing, Beam may divide the output file into numerous sections (e.g., {output.txt-00000-of-00001}), which the script reads and outputs the content of to confirm the transformation. This illustrates Beam's capacity to manage complex data processing jobs using straightforward but efficient code.

Transforms Principles in Beam

In Apache Beam, the PCollection object represents a collection of data to be processed. To start working with data, you first ingest it into a PCollection by using a read operation, which is itself a transform. This transform reads data from various sources, such as CSV files, databases, or other storage systems.

Read Operation: This is the initial step where data is ingested into a PCollection. The Read transform reads from a specified data source (e.g., CSV file) and creates a PCollection.

Apply Transformations: Once the data is in a PCollection, you can apply various transforms to process it. Transforms are operations that modify or analyze the data, such as filtering, mapping, or aggregating.

 
transformed_data = input_data | 'MapToUppercase' >> beam.Map(lambda x: x.upper())   

Write Operation: After applying transformations, you typically write the results to an output sink, such as a file or database.

By following to these guidelines, you can create a pipeline for data processing in which every action (read, transform, write) is represented as a transform that is employed on a PCollection. Workflows for processing data can be made flexible and scalable with this modular approach.

Example

Here's a comprehensive example that shows how to use Apache Beam to read data from a CSV file, transform it, and write the result. The steps involved in this example are reading a CSV file, changing every line to uppercase, and then writing the updated lines to an output file.

First, create a sample CSV file named data.csv

 
# Create a sample CSV file
with open('data.csv', 'w') as f:
    f.write("hello world\n")
    f.write("apache beam\n")
    f.write("data processing\n")
    f.write("pipeline example\n")
Pipeline Code
!pip install apache-beam

import apache_beam as beam
def transform_data(element):
    return element.upper()
with beam.Pipeline() as pipeline:
    # Read data from a CSV file
    input_data = pipeline | 'ReadFromCSV' >> beam.io.ReadFromText('data.csv')
    
    transformed_data = input_data | 'MapToUppercase' >> beam.Map(transform_data)
    
    # Write the transformed data to an output text file
    transformed_data | 'WriteToText' >> beam.io.WriteToText('output.txt')

# Run the pipeline and wait until it's done
result = pipeline.run()
result.wait_until_finish()

# Read and print the results from the output file
with open('output.txt-00000-of-00001', 'r') as f:
    output = f.read()

print(output)
Read Data

input_data = pipeline | 'ReadFromCSV' >> beam.io.ReadFromText('data.csv')

Transform Data
transformed_data = input_data | 'MapToUppercase' >> beam.Map(transform_data)

Write Data
transformed_data | 'WriteToText' >> beam.io.WriteToText('output.txt')   

Run the Pipeline:

 
result = pipeline.run()
result.wait_until_finish()   

Print the Output:

 
with open('output.txt-00000-of-00001', 'r') as f:
    output = f.read()
print(output)   

Output

The output file output.txt-00000-of-00001 will contain:

 
HELLO WORLD
APACHE BEAM
DATA PROCESSING
PIPELINE EXAMPLE

Explanation

The code demonstrates an Apache Beam pipeline that reads from a CSV file, transforms each line to uppercase, and writes the results to an output file. It starts by creating a sample `data.csv` file. The pipeline begins with the `ReadFromCSV` transform to read the CSV data into a `PCollection`. It then applies the `MapToUppercase` transform using `beam.Map` to convert each line to uppercase. The transformed data is written to `output.txt` with the `WriteToText` transform. The pipeline is executed and waits for completion using `pipeline.run()`. Finally, the output file is read, and the results are printed, demonstrating the essential steps of a Beam pipeline: reading, transforming, and writing data.

Apache Beam

Using batch or stream-based integrations, Apache Beam is an open-source software development kit that lets you create various data pipelines and run them either directly or indirectly. Each pipeline allows you to add different transformations. However, Beam's true strength lies in its platform independence, which is derived from its lack of reliance on any particular compute engine. You specify which "runner" to apply in order to calculate the transformation. Defaulting to your local computer resource, you can select a different one, like Cloud Dataflow or a Spark engine.

Installation

To run this example, you need to have Apache Beam installed. You can install it via pip if you're using python

Example

 
import apache_beam as beam

def split_words(line):
    """Splits a line of text into individual words."""
    return line.split()

def count_words(elements):
    """Counts the frequency of each word in the list of words."""
    word_counts = {}
    for word in elements:
        word_counts[word] = word_counts.get(word, 0) + 1
    return word_counts.items()

def format_result(word_count):
    """Formats the result for output."""
    word, count = word_count
    return f'{word}: {count}'

def run():
    # Define the pipeline
    with beam.Pipeline() as pipeline:
        # Read input data from a file
        lines = (
            pipeline
            | 'Read' >> beam.io.ReadFromText('input.txt')
        )
        
        # Split lines into words
        words = (
            lines
            | 'Split' >> beam.FlatMap(split_words)
        )
        
        # Count the words
        word_counts = (
            words
            | 'PairWithOne' >> beam.Map(lambda word: (word, 1))
            | 'GroupAndSum' >> beam.CombinePerKey(sum)
        )
        
        # Format the results for output
        formatted_results = (
            word_counts
            | 'FormatResults' >> beam.Map(format_result)
        )
        
        # Write results to an output file
        formatted_results | 'Write' >> beam.io.WriteToText('output.txt')

if __name__ == '__main__':
    run()   

Output:

 
hello: 2
world: 2
beam: 2

Explanation

The Apache Beam script performs a word count operation on text data. It starts by reading lines from an `input.txt` file and splits each line into words using the `split_words` function. It then maps each word to a key-value pair `(word, 1)`, followed by grouping these pairs and summing the counts using `beam.CombinePerKey(sum)`. The results are formatted into readable strings with the `format_result` function. Finally, the formatted word counts are written to `output.txt`. This pipeline processes text, counts word occurrences, and saves the results, handling data in a distributed manner if executed on platforms like Apache Flink or Google Cloud Dataflow.

Conclusion

In conclusion we have covered how to create data pipelines using Apache Beam, concentrating on a basic example of taking data from a CSV file, processing it, and publishing the results to an output file. We went over the fundamentals of building a pipeline, including how to use `Read} for data ingestion, `Map} for transformation, and `Write` for output. The procedure demonstrates how Beam's modular design facilitates the effective management of large-scale data processing. We demonstrated Beam's adaptability and strength in carrying out distributed data activities with understandable and readable code by executing the pipeline and managing the outcomes.

Next TopicHow to read bytes as stream in python 3

← prev next →