Apache Beam: ParDo

Overview

If you want to apply a function which transforms each element in an input collection, you should use the ParDo transform.

When You Should Use the ParDo Transform

Any time you want to perform some generic processing function on each element of a PCollection.

How to Use the ParDo Transform

To perform processing on an element using the ParDo transform, all you need is a Java class which is a subclass of the DoFn abstract class which contains your transformation logic. Your subclass must override the @processElement function with the transformation code (see below examples).

Example: Filter Word Counts with Max Length

  // The input PCollection to ParDo.
  PCollection<String> words = ...;

  // Apply a ParDo that takes maxWordLengthCutOffView as a side input.
  PCollection<String> wordsBelowCutOff =
  words.apply(ParDo
      .of(new DoFn<String, String>() {
          public void processElement(ProcessContext c) {
            String word = c.element();
            int maxLength = 10;
            if (word.length() <= maxLength) {
              c.output(word);
            }
          }
      })
  );

Conclusion

Check out other useful transforms from the official Apache Beam documentation.

Apache Beam: ParDo

Comments

Apache Beam and Google Cloud Dataflow

Apache Beam: KvSwap

More from this blog

Harnessing the IP Address  Data Type in Apache Druid (2025)

Geospatial Queries in Apache Druid: A Ride-Sharing Example

Automating Google Cloud with GitHub Actions using gcloud CLI

How to Run a Scheduled Bash Script with Google Cloud Platform

Google Dataflow Optimization: Streaming Engine

Overview

When You Should Use the ParDo Transform

How to Use the ParDo Transform

Example: Filter Word Counts with Max Length

Conclusion

Command Palette

Comments

Apache Beam and Google Cloud Dataflow

Apache Beam: KvSwap

More from this blog

Overview

When You Should Use the ParDo Transform

How to Use the ParDo Transform

Example: Filter Word Counts with Max Length

Conclusion