For the love of Data
Everything you need to ingest, store, and analyze billions of IPv4 & IPv6 addresses at interactive speed. Why a native IP data type? Storing every address as a plain string inflates segment size, hurts bitmap selectivity, and forces costly runtime ...
Apache Druid is a high-performance, real-time analytics database designed for large-scale data processing. Among its powerful features is the ability to handle geospatial data, enabling fast and efficient queries over latitude and longitude coordinat...
In today's DevOps landscape, automating tasks and deployments is essential for efficiency and reliability. One powerful way to achieve this is by leveraging GitHub Actions to connect with Google Cloud Platform (GCP) using the gcloud CLI. In this post...
Overview Why would you want to run a scheduled Bash script in the cloud? Couldn’t you just run cron on your local machine? Running a bash script in the cloud rather than on a local machine offers developers a multitude of advantages that can enhance ...
What is Streaming Engine "By default, the Dataflow pipeline runner executes the steps of your streaming pipeline entirely on worker virtual machines, consuming worker CPU, memory, and Persistent Disk storage. Dataflow's Streaming Engine moves pipelin...
What are Windows? Windows are a way to group your data by their event times. But, why do you want to group on time? So you can apply aggregations! An example might be if you have a stream of analytic data coming from mobile phones and want to count t...
Overview What if you only care about the values of your PCollection and not necessarily the keys? Maybe you have KV pairs for words and their counts in a string (KV<String, Integer>) and want to use the counts in another PCollection. You should use t...
Overview If you need to convert a PCollection into a KV pair using dynamically generated keys derived from the input element, you should check out the WithKeys transform. When You Should Use the WithKeys Transform When you only want to convert a PCol...
Overview If you want to convert every input element from a PCollection to a string, you should check out the ToString transforms. It can do everything from simply converting an object to a string by implicitly calling its toString() method to concate...