Apache Beam: Filter

Overview

Have you ever wanted to filter data from a PCollection?

With the Filter transform, you can!

Given a function that takes an input element and returns True or False, the Filter transform will only allow elements that return True to proceed through the pipeline and to the next transform. Also, if the elements of the input PCollection are Java Objects that are Comparable, you can use built-in Filters which will filter out elements based on the elements' natural ordering (see below for examples).

When You Should Use the Filter Transform

You should use the Filter transform whenever you need to clean a PCollection based on certain criteria.

How to Use the Filter Transform

Using this transform is quite easy. All you need is a function which returns a boolean (which includes lambda functions) for whether you want to keep the input element (True) or discard the element (False). It will then allow all elements which returned True to go to the next transform in the pipeline.

Example: Filtering with a function

PCollection<String> allStrings = Create.of("Hello", "world", "hi");
PCollection<String> longStrings = allStrings
    .apply(Filter.by(new SerializableFunction<String, Boolean>() {
      @Override
      public Boolean apply(String input) {
        return input.length() > 3;
      }
    }));

Example: Filtering with a Builtin Filter

PCollection<Long> numbers = Create.of(2L, 3L, 4L, 5L);
PCollection<Long> bigNumbers = numbers.apply(Filter.greaterThan(3));
PCollection<Long> smallNumbers = numbers.apply(Filter.lessThanEq(3));

Example: Filtering with a Lambda

PCollection<Long> numbers = Create.of(2L, 3L, 4L, 5L);
PCollection<Long> bigNumbers = numbers.apply(Filter.by(number -> number > 3));

Conclusion

Instead of writing a custom Do Function to filter elements in a PCollection, you should use the Filter Transform.

Check out other useful transforms from the official Apache Beam documentation.

Apache Beam: Filter

Comments

Apache Beam and Google Cloud Dataflow

Apache Beam: MapElements and FlatMapElements Transforms

More from this blog

Harnessing the IP Address  Data Type in Apache Druid (2025)

Geospatial Queries in Apache Druid: A Ride-Sharing Example

Automating Google Cloud with GitHub Actions using gcloud CLI

How to Run a Scheduled Bash Script with Google Cloud Platform

Google Dataflow Optimization: Streaming Engine

Overview

When You Should Use the Filter Transform

How to Use the Filter Transform

Example: Filtering with a function

Example: Filtering with a Builtin Filter

Example: Filtering with a Lambda

Conclusion

Command Palette

Comments

Apache Beam and Google Cloud Dataflow

Apache Beam: MapElements and FlatMapElements Transforms

More from this blog

Overview

When You Should Use the Filter Transform

How to Use the Filter Transform

Example: Filtering with a function

Example: Filtering with a Builtin Filter

Example: Filtering with a Lambda

Conclusion