<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Nikhil Rao's Blog]]></title><description><![CDATA[For the love of Data]]></description><link>https://nikhilrao.blog</link><generator>RSS for Node</generator><lastBuildDate>Fri, 13 Nov 2026 10:24:51 GMT</lastBuildDate><atom:link href="https://nikhilrao.blog/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Harnessing the IP Address  Data Type in Apache Druid (2025)]]></title><description><![CDATA[Everything you need to ingest, store, and analyze billions of IPv4 & IPv6 addresses at interactive speed. Storing every address as a plain string inflates…]]></description><link>https://nikhilrao.blog/harnessing-the-ip-address-data-type-in-apache-druid-2025</link><guid isPermaLink="true">https://nikhilrao.blog/harnessing-the-ip-address-data-type-in-apache-druid-2025</guid><category><![CDATA[Druid]]></category><category><![CDATA[apachedruid]]></category><category><![CDATA[apache druid]]></category><category><![CDATA[ip address]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Sun, 18 May 2025 01:30:38 GMT</pubDate><content:encoded><![CDATA[<blockquote>
<p><em>Everything you need to ingest, store, and analyze billions of IPv4 &amp; IPv6 addresses at interactive speed.</em></p>
</blockquote>
<hr />
<h2 id="heading-why-a-native-ip-data-type"><strong>Why a native IP data type?</strong></h2>
<p>Storing every address as a plain string inflates segment size, hurts bitmap selectivity, and forces costly runtime casts for subnet math.</p>
<p>The complex ipAddress (128‑bit) and ipPrefix (136‑bit) encodings introduced in Druid 30 collapse IPv4 and IPv6 into compact binaries, enabling index‑friendly comparisons, hash‑based GROUP BY, and true subnet pruning down to the segment level.</p>
<hr />
<h2 id="heading-enabling-the-feature"><strong>Enabling the feature</strong></h2>
<pre><code class="lang-bash"><span class="hljs-comment"># common.runtime.properties</span>
druid.extensions.loadList=[<span class="hljs-string">"imply-utility-belt"</span>]   <span class="hljs-comment"># ships the IP complex type</span>
</code></pre>
<p><em>IPv6 support is still</em> <strong><em>beta</em></strong>*—keep an eye on future releases before turning it on in production.*</p>
<hr />
<h2 id="heading-designing-the-ingestion-spec"><strong>Designing the ingestion spec</strong></h2>
<pre><code class="lang-json"><span class="hljs-string">"dimensionsSpec"</span>: {
  <span class="hljs-attr">"dimensions"</span>: [
    <span class="hljs-string">"page"</span>,
    <span class="hljs-string">"language"</span>,
    { <span class="hljs-attr">"type"</span>: <span class="hljs-string">"ipAddress"</span>, <span class="hljs-attr">"name"</span>: <span class="hljs-string">"remote_address"</span> },  <span class="hljs-comment">// single host</span>
    { <span class="hljs-attr">"type"</span>: <span class="hljs-string">"ipPrefix"</span>,  <span class="hljs-attr">"name"</span>: <span class="hljs-string">"network_prefix"</span> }   <span class="hljs-comment">// CIDR block</span>
  ]
}
</code></pre>
<p>ipAddress stores 128‑bit hosts, while ipPrefix keeps the host bits <strong>plus</strong> an 8‑bit prefix length so range look‑ups remain constant‑time.</p>
<h3 id="heading-example-ingestion-spec-with-data">Example Ingestion Spec with Data</h3>
<p>An example ingestion spec with inline data is shown below</p>
<pre><code class="lang-json">{
  <span class="hljs-attr">"type"</span>: <span class="hljs-string">"index_parallel"</span>,
  <span class="hljs-attr">"spec"</span>: {
    <span class="hljs-attr">"dataSchema"</span>: {
      <span class="hljs-attr">"dataSource"</span>: <span class="hljs-string">"inline_data_ip_address"</span>,
      <span class="hljs-attr">"timestampSpec"</span>: {
        <span class="hljs-attr">"column"</span>: <span class="hljs-string">"timestamp"</span>,
        <span class="hljs-attr">"format"</span>: <span class="hljs-string">"iso"</span>,
        <span class="hljs-attr">"missingValue"</span>: <span class="hljs-literal">null</span>
      },
      <span class="hljs-attr">"dimensionsSpec"</span>: {
        <span class="hljs-attr">"dimensions"</span>: [
          {
            <span class="hljs-attr">"type"</span>: <span class="hljs-string">"string"</span>,
            <span class="hljs-attr">"name"</span>: <span class="hljs-string">"page"</span>,
            <span class="hljs-attr">"multiValueHandling"</span>: <span class="hljs-string">"SORTED_ARRAY"</span>,
            <span class="hljs-attr">"createBitmapIndex"</span>: <span class="hljs-literal">true</span>
          },
          {
            <span class="hljs-attr">"type"</span>: <span class="hljs-string">"string"</span>,
            <span class="hljs-attr">"name"</span>: <span class="hljs-string">"language"</span>,
            <span class="hljs-attr">"multiValueHandling"</span>: <span class="hljs-string">"SORTED_ARRAY"</span>,
            <span class="hljs-attr">"createBitmapIndex"</span>: <span class="hljs-literal">true</span>
          },
          {
            <span class="hljs-attr">"type"</span>: <span class="hljs-string">"ipAddress"</span>,
            <span class="hljs-attr">"name"</span>: <span class="hljs-string">"remote_address"</span>,
            <span class="hljs-attr">"multiValueHandling"</span>: <span class="hljs-string">"SORTED_ARRAY"</span>,
            <span class="hljs-attr">"createBitmapIndex"</span>: <span class="hljs-literal">true</span>
          },
          {
            <span class="hljs-attr">"type"</span>: <span class="hljs-string">"ipPrefix"</span>,
            <span class="hljs-attr">"name"</span>: <span class="hljs-string">"network_prefix"</span>,
            <span class="hljs-attr">"multiValueHandling"</span>: <span class="hljs-string">"SORTED_ARRAY"</span>,
            <span class="hljs-attr">"createBitmapIndex"</span>: <span class="hljs-literal">true</span>
          }
        ],
        <span class="hljs-attr">"dimensionExclusions"</span>: [
          <span class="hljs-string">"__time"</span>,
          <span class="hljs-string">"timestamp"</span>
        ],
        <span class="hljs-attr">"includeAllDimensions"</span>: <span class="hljs-literal">false</span>,
        <span class="hljs-attr">"useSchemaDiscovery"</span>: <span class="hljs-literal">false</span>
      },
      <span class="hljs-attr">"metricsSpec"</span>: [],
      <span class="hljs-attr">"granularitySpec"</span>: {
        <span class="hljs-attr">"type"</span>: <span class="hljs-string">"uniform"</span>,
        <span class="hljs-attr">"segmentGranularity"</span>: <span class="hljs-string">"DAY"</span>,
        <span class="hljs-attr">"queryGranularity"</span>: {
          <span class="hljs-attr">"type"</span>: <span class="hljs-string">"none"</span>
        },
        <span class="hljs-attr">"rollup"</span>: <span class="hljs-literal">false</span>,
        <span class="hljs-attr">"intervals"</span>: []
      },
      <span class="hljs-attr">"transformSpec"</span>: {
        <span class="hljs-attr">"filter"</span>: <span class="hljs-literal">null</span>,
        <span class="hljs-attr">"transforms"</span>: []
      }
    },
    <span class="hljs-attr">"ioConfig"</span>: {
      <span class="hljs-attr">"type"</span>: <span class="hljs-string">"index_parallel"</span>,
      <span class="hljs-attr">"inputSource"</span>: {
        <span class="hljs-attr">"type"</span>: <span class="hljs-string">"inline"</span>,
        <span class="hljs-attr">"data"</span>: <span class="hljs-string">"timestamp,page,language,remote_address,network_prefix\n2025-05-17T00:00:00,/index.html,en,bc89:60a9:23b8:c1e9:3924:56de:3eb1:3b90,bc89:60a9:23b8::/48\n2025-05-17T00:05:00,/index.html,jp,108.3.17.153,108.3.17.0/24\n2025-05-17T00:10:00,/,en,59.143.170.24,59.143.170.0/24\n2025-05-17T00:15:00,/checkout,jp,50.231.6.41,50.231.6.0/24\n2025-05-17T00:20:00,/checkout,de,150.218.29.172,150.218.29.0/24\n2025-05-17T00:25:00,/login,en,571a:a876:6c30:7511:b2b9:437a:28df:6ec4,571a:a876:6c30::/48\n2025-05-17T00:30:00,/login,es,195.116.89.238,195.116.89.0/24\n2025-05-17T00:35:00,/account,en,24.194.103.151,24.194.103.0/24\n2025-05-17T00:40:00,/account,fr,759c:de66:bacf:b3d0:b1f:9163:ce9f:f57f,759c:de66:bacf::/48\n2025-05-17T00:45:00,/checkout,en,4b0d:bb41:8d52:88f1:142c:3fe8:60e7:a113,4b0d:bb41:8d52::/48\n"</span>
      },
      <span class="hljs-attr">"inputFormat"</span>: {
        <span class="hljs-attr">"type"</span>: <span class="hljs-string">"csv"</span>,
        <span class="hljs-attr">"findColumnsFromHeader"</span>: <span class="hljs-literal">true</span>
      },
      <span class="hljs-attr">"appendToExisting"</span>: <span class="hljs-literal">false</span>,
      <span class="hljs-attr">"dropExisting"</span>: <span class="hljs-literal">false</span>
    },
    <span class="hljs-attr">"tuningConfig"</span>: {
      <span class="hljs-attr">"type"</span>: <span class="hljs-string">"index_parallel"</span>,
      <span class="hljs-attr">"maxRowsPerSegment"</span>: <span class="hljs-number">5000000</span>,
      <span class="hljs-attr">"appendableIndexSpec"</span>: {
        <span class="hljs-attr">"type"</span>: <span class="hljs-string">"onheap"</span>,
        <span class="hljs-attr">"preserveExistingMetrics"</span>: <span class="hljs-literal">false</span>
      },
      <span class="hljs-attr">"maxRowsInMemory"</span>: <span class="hljs-number">1000000</span>,
      <span class="hljs-attr">"maxBytesInMemory"</span>: <span class="hljs-number">0</span>,
      <span class="hljs-attr">"skipBytesInMemoryOverheadCheck"</span>: <span class="hljs-literal">false</span>,
      <span class="hljs-attr">"maxTotalRows"</span>: <span class="hljs-literal">null</span>,
      <span class="hljs-attr">"numShards"</span>: <span class="hljs-literal">null</span>,
      <span class="hljs-attr">"splitHintSpec"</span>: <span class="hljs-literal">null</span>,
      <span class="hljs-attr">"partitionsSpec"</span>: {
        <span class="hljs-attr">"type"</span>: <span class="hljs-string">"dynamic"</span>,
        <span class="hljs-attr">"maxRowsPerSegment"</span>: <span class="hljs-number">5000000</span>,
        <span class="hljs-attr">"maxTotalRows"</span>: <span class="hljs-literal">null</span>
      },
      <span class="hljs-attr">"indexSpec"</span>: {
        <span class="hljs-attr">"bitmap"</span>: {
          <span class="hljs-attr">"type"</span>: <span class="hljs-string">"roaring"</span>
        },
        <span class="hljs-attr">"dimensionCompression"</span>: <span class="hljs-string">"lz4"</span>,
        <span class="hljs-attr">"stringDictionaryEncoding"</span>: {
          <span class="hljs-attr">"type"</span>: <span class="hljs-string">"utf8"</span>
        },
        <span class="hljs-attr">"metricCompression"</span>: <span class="hljs-string">"lz4"</span>,
        <span class="hljs-attr">"longEncoding"</span>: <span class="hljs-string">"longs"</span>
      },
      <span class="hljs-attr">"indexSpecForIntermediatePersists"</span>: {
        <span class="hljs-attr">"bitmap"</span>: {
          <span class="hljs-attr">"type"</span>: <span class="hljs-string">"roaring"</span>
        },
        <span class="hljs-attr">"dimensionCompression"</span>: <span class="hljs-string">"lz4"</span>,
        <span class="hljs-attr">"stringDictionaryEncoding"</span>: {
          <span class="hljs-attr">"type"</span>: <span class="hljs-string">"utf8"</span>
        },
        <span class="hljs-attr">"metricCompression"</span>: <span class="hljs-string">"lz4"</span>,
        <span class="hljs-attr">"longEncoding"</span>: <span class="hljs-string">"longs"</span>
      },
      <span class="hljs-attr">"maxPendingPersists"</span>: <span class="hljs-number">0</span>,
      <span class="hljs-attr">"forceGuaranteedRollup"</span>: <span class="hljs-literal">false</span>,
      <span class="hljs-attr">"reportParseExceptions"</span>: <span class="hljs-literal">false</span>,
      <span class="hljs-attr">"pushTimeout"</span>: <span class="hljs-number">0</span>,
      <span class="hljs-attr">"segmentWriteOutMediumFactory"</span>: <span class="hljs-literal">null</span>,
      <span class="hljs-attr">"maxNumConcurrentSubTasks"</span>: <span class="hljs-number">1</span>,
      <span class="hljs-attr">"maxRetry"</span>: <span class="hljs-number">3</span>,
      <span class="hljs-attr">"taskStatusCheckPeriodMs"</span>: <span class="hljs-number">1000</span>,
      <span class="hljs-attr">"chatHandlerTimeout"</span>: <span class="hljs-string">"PT10S"</span>,
      <span class="hljs-attr">"chatHandlerNumRetries"</span>: <span class="hljs-number">5</span>,
      <span class="hljs-attr">"maxNumSegmentsToMerge"</span>: <span class="hljs-number">100</span>,
      <span class="hljs-attr">"totalNumMergeTasks"</span>: <span class="hljs-number">10</span>,
      <span class="hljs-attr">"logParseExceptions"</span>: <span class="hljs-literal">false</span>,
      <span class="hljs-attr">"maxParseExceptions"</span>: <span class="hljs-number">2147483647</span>,
      <span class="hljs-attr">"maxSavedParseExceptions"</span>: <span class="hljs-number">0</span>,
      <span class="hljs-attr">"maxColumnsToMerge"</span>: <span class="hljs-number">-1</span>,
      <span class="hljs-attr">"awaitSegmentAvailabilityTimeoutMillis"</span>: <span class="hljs-number">0</span>,
      <span class="hljs-attr">"maxAllowedLockCount"</span>: <span class="hljs-number">-1</span>,
      <span class="hljs-attr">"numPersistThreads"</span>: <span class="hljs-number">1</span>,
      <span class="hljs-attr">"partitionDimensions"</span>: []
    }
  },
  <span class="hljs-attr">"context"</span>: {
    <span class="hljs-attr">"forceTimeChunkLock"</span>: <span class="hljs-literal">true</span>,
    <span class="hljs-attr">"useLineageBasedSegmentAllocation"</span>: <span class="hljs-literal">true</span>
  },
  <span class="hljs-attr">"dataSource"</span>: <span class="hljs-string">"inline_data_ip_address"</span>
}
</code></pre>
<h3 id="heading-quick-sanity-check"><strong>Quick sanity check</strong></h3>
<pre><code class="lang-pgsql"><span class="hljs-keyword">SELECT</span> 
  IPV4_STRINGIFY(remote_address) <span class="hljs-keyword">AS</span> ip,
  IPV4_STRINGIFY(network_prefix) <span class="hljs-keyword">AS</span> <span class="hljs-type">cidr</span>
<span class="hljs-keyword">FROM</span> traffic <span class="hljs-keyword">LIMIT</span> <span class="hljs-number">5</span>
</code></pre>
<hr />
<h2 id="heading-querying-with-ipcentric-functions"><strong>Querying with IP‑centric functions</strong></h2>
<h3 id="heading-matching-amp-searching"><strong>Matching &amp; searching</strong></h3>
<pre><code class="lang-sql"><span class="hljs-comment">-- Is the client inside a zero‑trust subnet?</span>
<span class="hljs-keyword">SELECT</span> <span class="hljs-keyword">COUNT</span>(*) 
<span class="hljs-keyword">FROM</span> traffic 
<span class="hljs-keyword">WHERE</span> IPV4_MATCH(remote_address, <span class="hljs-string">'108.3.0.0/16'</span>)
</code></pre>
<p><code>IPV4_MATCH</code> needs an exact host or CIDR on the right.</p>
<hr />
<h2 id="heading-migrating-from-string-ip"><strong>Migrating from STRING → IP</strong></h2>
<ol>
<li><p><strong>Backfill</strong>: Re‑ingest recent partitions with the new dimensionSpec.</p>
</li>
<li><p><strong>Realtime streams</strong>: Add a <em>transform</em> when your parser emits the address:</p>
</li>
</ol>
<pre><code class="lang-json">{ <span class="hljs-attr">"type"</span>:<span class="hljs-string">"expression"</span>, <span class="hljs-attr">"name"</span>:<span class="hljs-string">"remote_address"</span>,
  <span class="hljs-attr">"expression"</span>:<span class="hljs-string">"IP_PARSE(remote_address_str)"</span> }
</code></pre>
<ol start="2">
<li><p><strong>Join strategy</strong>: For mixed historical segments, wrap filters in COALESCE(IP_PARSE(addr_str), addr_bin) so queries hit both encodings.</p>
</li>
<li><p>Once old segments age out, drop the legacy string column.</p>
</li>
</ol>
<hr />
<h2 id="heading-performance-tips-amp-caveats"><strong>Performance tips &amp; caveats</strong></h2>
<div class="hn-table">
<table>
<thead>
<tr>
<td><strong>Tip</strong></td><td><strong>Why</strong></td></tr>
</thead>
<tbody>
<tr>
<td><strong>Use ipPrefix for common subnet filters</strong></td><td>Bitmap indexes &amp; Bloom filters can prune segments before the broker fan‑out.</td></tr>
<tr>
<td><strong>Keep prefix lengths ≤ 48 for IPv6</strong></td><td>Finer masks can explode the prefix space and hurt pruning.</td></tr>
<tr>
<td><strong>Disable dictionary encoding on bulk host columns</strong> (dimensionCompression: "none")</td><td>When cardinality ≈ row‑count, front‑load encoding cost is wasteful.</td></tr>
<tr>
<td><strong>Be mindful of JSON serialization</strong></td><td>Result sets with ipAddress columns emit the binary as Base64; wrap with <code>IPV4_STRINGIFY</code> for dashboards.</td></tr>
<tr>
<td><strong>Feature maturity</strong></td><td>IPv6 operations are marked <em>beta</em>—benchmark with real workloads before committing.</td></tr>
</tbody>
</table>
</div><hr />
<h2 id="heading-wrapping-up"><strong>Wrapping up</strong></h2>
<p>The native IP data types radically simplify network analytics in Druid:</p>
<ul>
<li><p><strong>Smaller segments</strong> → lower storage and faster scans.</p>
</li>
<li><p><strong>True subnet intelligence</strong> → no more ad‑hoc regex parsing.</p>
</li>
<li><p><strong>Full SQL ergonomics</strong> with first‑class GROUP BY and WHERE support.</p>
</li>
</ul>
<p>Whether you’re protecting an edge, feeding a SIEM, or building click‑stream geo dashboards, upgrading to the ipAddress/ipPrefix complex types unlocks millisecond‑level insights at cloud scale.</p>
<p>Happy querying! 🐉✨</p>
]]></content:encoded></item><item><title><![CDATA[Geospatial Queries in Apache Druid: A Ride-Sharing Example]]></title><description><![CDATA[Apache Druid is a high-performance, real-time analytics database designed for large-scale data processing. Among its powerful features is the ability to…]]></description><link>https://nikhilrao.blog/geospatial-queries-in-apache-druid-a-ride-sharing-example</link><guid isPermaLink="true">https://nikhilrao.blog/geospatial-queries-in-apache-druid-a-ride-sharing-example</guid><category><![CDATA[apache druid]]></category><category><![CDATA[Druid]]></category><category><![CDATA[Geospatial]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Wed, 14 May 2025 01:01:02 GMT</pubDate><content:encoded><![CDATA[<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1747184382196/79dbdaaf-abe2-477b-af60-830ee5fcbc12.jpeg" alt class="image--center mx-auto" /></p>
<p>Apache Druid is a high-performance, real-time analytics database designed for large-scale data processing. Among its powerful features is the ability to handle geospatial data, enabling fast and efficient queries over latitude and longitude coordinates. In this post, we’ll explore a practical scenario where geospatial queries in Druid shine: real-time analytics for a ride-sharing platform. We’ll walk through the use case, provide a sample query, and highlight why Druid is a great fit for geospatial workloads.</p>
<h2 id="heading-the-scenario-real-time-ride-sharing-analytics">The Scenario: Real-Time Ride-Sharing Analytics</h2>
<p>Imagine a ride-sharing company that operates in a bustling city like New York. Every day, the platform processes millions of events, including driver location updates and passenger ride requests. A critical operational need is to match passengers with the nearest available driver in real time. For example, when a passenger requests a ride, the system must identify all drivers within a 10-kilometer radius of the pickup location and assign the closest one—all within milliseconds.</p>
<p>This is where geospatial data comes into play. Driver locations are continuously ingested as latitude and longitude coordinates, and the system needs to perform proximity searches to find drivers near a passenger’s coordinates. Apache Druid’s geospatial query capabilities make this possible at scale.</p>
<h2 id="heading-setting-up-the-data-in-druid">Setting Up the Data in Druid</h2>
<p>Let’s assume the ride-sharing company stores driver location data in a Druid datasource called <code>driver_locations</code>. The datasource has the following columns:</p>
<ul>
<li><p><code>driver_id</code>: Unique identifier for each driver.</p>
</li>
<li><p><code>latitude</code>: The driver’s latitude coordinate.</p>
</li>
<li><p><code>longitude</code>: The driver’s longitude coordinate.</p>
</li>
<li><p><code>__time</code>: The timestamp of the location update.</p>
</li>
</ul>
<p>The data is ingested in real time as drivers move, ensuring the system always has fresh location information. Druid’s columnar storage and indexing optimize this dataset for fast queries, even with millions of rows.</p>
<h2 id="heading-ingesting-geospatial-data-in-druid">Ingesting Geospatial Data in Druid</h2>
<p>To enable geospatial queries, the ride-sharing company must first ingest driver location data into the <code>driver_locations</code> datasource. For this example, we’ll use an inline ingestion spec to load a small dataset directly, which is useful for testing or demonstration purposes.</p>
<p>Here’s how the ingestion process works:</p>
<ul>
<li><p><strong>Data Source</strong>: Driver location updates are provided as CSV data within the ingestion spec. Each row contains <code>driver_id</code>, <code>latitude</code>, <code>longitude</code>, and <code>timestamp</code>. For example, the inline sample dataset below includes a few driver locations in New York City.</p>
</li>
<li><p><strong>Ingestion Spec</strong>: The company defines a Druid ingestion spec with an inline data source in CSV format. The spec maps the CSV columns to the <code>driver_locations</code> datasource, with <code>timestamp</code> as the <code>__time</code> column, <code>latitude</code> and <code>longitude</code> as double dimensions, and a spatial dimension for geospatial queries. Here’s the ingestion spec:</p>
<pre><code class="lang-json">  {
    <span class="hljs-attr">"type"</span>: <span class="hljs-string">"index_parallel"</span>,
    <span class="hljs-attr">"spec"</span>: {
      <span class="hljs-attr">"dataSchema"</span>: {
        <span class="hljs-attr">"dataSource"</span>: <span class="hljs-string">"driver_locations"</span>,
        <span class="hljs-attr">"timestampSpec"</span>: {
          <span class="hljs-attr">"column"</span>: <span class="hljs-string">"timestamp"</span>,
          <span class="hljs-attr">"format"</span>: <span class="hljs-string">"iso"</span>
        },
        <span class="hljs-attr">"dimensionsSpec"</span>: {
          <span class="hljs-attr">"dimensions"</span>: [
            {
              <span class="hljs-attr">"type"</span>: <span class="hljs-string">"string"</span>,
              <span class="hljs-attr">"name"</span>: <span class="hljs-string">"driver_id"</span>,
              <span class="hljs-attr">"multiValueHandling"</span>: <span class="hljs-string">"SORTED_ARRAY"</span>,
              <span class="hljs-attr">"createBitmapIndex"</span>: <span class="hljs-literal">true</span>
            },
            {
              <span class="hljs-attr">"type"</span>: <span class="hljs-string">"double"</span>,
              <span class="hljs-attr">"name"</span>: <span class="hljs-string">"latitude"</span>,
              <span class="hljs-attr">"multiValueHandling"</span>: <span class="hljs-string">"SORTED_ARRAY"</span>,
              <span class="hljs-attr">"createBitmapIndex"</span>: <span class="hljs-literal">false</span>
            },
            {
              <span class="hljs-attr">"type"</span>: <span class="hljs-string">"double"</span>,
              <span class="hljs-attr">"name"</span>: <span class="hljs-string">"longitude"</span>,
              <span class="hljs-attr">"multiValueHandling"</span>: <span class="hljs-string">"SORTED_ARRAY"</span>,
              <span class="hljs-attr">"createBitmapIndex"</span>: <span class="hljs-literal">false</span>
            },
            {
              <span class="hljs-attr">"type"</span>: <span class="hljs-string">"spatial"</span>,
              <span class="hljs-attr">"name"</span>: <span class="hljs-string">"coordinates"</span>,
              <span class="hljs-attr">"dims"</span>: [<span class="hljs-string">"latitude"</span>, <span class="hljs-string">"longitude"</span>],
              <span class="hljs-attr">"multiValueHandling"</span>: <span class="hljs-string">"SORTED_ARRAY"</span>,
              <span class="hljs-attr">"createBitmapIndex"</span>: <span class="hljs-literal">true</span>
            }
          ],
          <span class="hljs-attr">"dimensionExclusions"</span>: [<span class="hljs-string">"__time"</span>, <span class="hljs-string">"timestamp"</span>],
          <span class="hljs-attr">"includeAllDimensions"</span>: <span class="hljs-literal">false</span>,
          <span class="hljs-attr">"useSchemaDiscovery"</span>: <span class="hljs-literal">false</span>
        },
        <span class="hljs-attr">"metricsSpec"</span>: [],
        <span class="hljs-attr">"granularitySpec"</span>: {
          <span class="hljs-attr">"type"</span>: <span class="hljs-string">"uniform"</span>,
          <span class="hljs-attr">"segmentGranularity"</span>: <span class="hljs-string">"DAY"</span>,
          <span class="hljs-attr">"queryGranularity"</span>: {
            <span class="hljs-attr">"type"</span>: <span class="hljs-string">"none"</span>
          },
          <span class="hljs-attr">"rollup"</span>: <span class="hljs-literal">false</span>
        }
      },
      <span class="hljs-attr">"ioConfig"</span>: {
        <span class="hljs-attr">"type"</span>: <span class="hljs-string">"index_parallel"</span>,
        <span class="hljs-attr">"inputSource"</span>: {
          <span class="hljs-attr">"type"</span>: <span class="hljs-string">"inline"</span>,
          <span class="hljs-attr">"data"</span>: <span class="hljs-string">"driver_id,latitude,longitude,timestamp\nD123,40.7128,-74.0060,2025-05-13T17:41:00Z\nD124,40.7210,-74.0050,2025-05-13T17:41:05Z\nD125,40.7050,-74.0080,2025-05-13T17:41:10Z"</span>
        },
        <span class="hljs-attr">"inputFormat"</span>: {
          <span class="hljs-attr">"type"</span>: <span class="hljs-string">"csv"</span>,
          <span class="hljs-attr">"findColumnsFromHeader"</span>: <span class="hljs-literal">true</span>
        },
        <span class="hljs-attr">"appendToExisting"</span>: <span class="hljs-literal">false</span>,
        <span class="hljs-attr">"dropExisting"</span>: <span class="hljs-literal">false</span>
      },
      <span class="hljs-attr">"tuningConfig"</span>: {
        <span class="hljs-attr">"type"</span>: <span class="hljs-string">"index_parallel"</span>,
        <span class="hljs-attr">"maxRowsPerSegment"</span>: <span class="hljs-number">5000000</span>,
        <span class="hljs-attr">"partitionsSpec"</span>: {
          <span class="hljs-attr">"type"</span>: <span class="hljs-string">"dynamic"</span>,
          <span class="hljs-attr">"maxRowsPerSegment"</span>: <span class="hljs-number">5000000</span>
        },
        <span class="hljs-attr">"indexSpec"</span>: {
          <span class="hljs-attr">"bitmap"</span>: {
            <span class="hljs-attr">"type"</span>: <span class="hljs-string">"roaring"</span>
          },
          <span class="hljs-attr">"dimensionCompression"</span>: <span class="hljs-string">"lz4"</span>,
          <span class="hljs-attr">"stringDictionaryEncoding"</span>: {
            <span class="hljs-attr">"type"</span>: <span class="hljs-string">"utf8"</span>
          },
          <span class="hljs-attr">"metricCompression"</span>: <span class="hljs-string">"lz4"</span>,
          <span class="hljs-attr">"longEncoding"</span>: <span class="hljs-string">"longs"</span>
        }
      }
    }
  }
</code></pre>
</li>
<li><p><strong>Spatial Indexing</strong>: The <code>dimensionsSpec</code> includes a <code>spatial</code> dimension named <code>coordinates</code> that combines <code>latitude</code> and <code>longitude</code>, enabling efficient geospatial queries.</p>
</li>
<li><p><strong>Batch Processing</strong>: The inline spec processes the provided CSV data as a one-time batch using parallel indexing, ideal for testing geospatial queries with a small dataset before scaling to real-time ingestion.</p>
</li>
</ul>
<p>This setup allows the ride-sharing platform to ingest a sample dataset and prepare it for geospatial querying, leveraging Druid’s spatial indexing for performance.</p>
<h2 id="heading-querying-geospatial-data-in-druid">Querying Geospatial Data in Druid</h2>
<p>To match a passenger with nearby drivers, the company needs to query the <code>driver_locations</code> datasource for drivers within a 10-kilometer radius of the passenger’s pickup point. For this example, let’s say the passenger is at coordinates (40.7128, -74.0060), roughly the location of downtown Manhattan.</p>
<p>Druid supports geospatial queries through its native query language, using a <code>scan</code> query with a spatial filter. Here’s the query to find drivers near the passenger:</p>
<pre><code class="lang-json">{
  <span class="hljs-attr">"queryType"</span>: <span class="hljs-string">"scan"</span>,
  <span class="hljs-attr">"dataSource"</span>: {
    <span class="hljs-attr">"type"</span>: <span class="hljs-string">"table"</span>,
    <span class="hljs-attr">"name"</span>: <span class="hljs-string">"driver_locations"</span>
  },
  <span class="hljs-attr">"intervals"</span>: {
    <span class="hljs-attr">"type"</span>: <span class="hljs-string">"intervals"</span>,
    <span class="hljs-attr">"intervals"</span>: [
      <span class="hljs-string">"2024-05-13T17:31:00.000Z/2026-05-13T17:42:00.000Z"</span>
    ]
  },
  <span class="hljs-attr">"resultFormat"</span>: <span class="hljs-string">"compactedList"</span>,
  <span class="hljs-attr">"limit"</span>: <span class="hljs-number">1000</span>,
  <span class="hljs-attr">"columns"</span>: [
    <span class="hljs-string">"driver_id"</span>,
    <span class="hljs-string">"coordinates"</span>
  ],
  <span class="hljs-attr">"granularity"</span>: {
    <span class="hljs-attr">"type"</span>: <span class="hljs-string">"all"</span>
  },
  <span class="hljs-attr">"filter"</span>: {
    <span class="hljs-attr">"type"</span>: <span class="hljs-string">"spatial"</span>,
    <span class="hljs-attr">"dimension"</span>: <span class="hljs-string">"coordinates"</span>,
    <span class="hljs-attr">"bound"</span>: {
      <span class="hljs-attr">"type"</span>: <span class="hljs-string">"radius"</span>,
      <span class="hljs-attr">"coords"</span>: [
        <span class="hljs-number">40.6678</span>,
        <span class="hljs-number">-74.051</span>
      ],
      <span class="hljs-attr">"radius"</span>: <span class="hljs-number">10</span>,
      <span class="hljs-attr">"radiusUnit"</span>: <span class="hljs-string">"kilometers"</span>
    }
  }
}
</code></pre>
<p>Let’s break down the query:</p>
<ul>
<li><p><strong>Time Filter</strong>: The <code>intervals</code> field restricts the query to data from the last 10 minutes (e.g., from 17:31 to 17:41 on May 13, 2025), ensuring only recent driver locations are considered for real-time matching.</p>
</li>
<li><p><strong>Geospatial Filter</strong>: The <code>filter</code> applies a <code>spatial</code> condition on the <code>coordinates</code> dimension, using a radius bound. The <code>coords</code> (40.6678, -74.0510), <code>radius</code> 10, and <code>radiusUnit</code> “kilometers” define a circle around the passenger’s location (40.7128, -74.0060), approximating a 10-kilometer radius.</p>
</li>
<li><p><strong>Columns</strong>: The query returns <code>__time</code>, <code>driver_id</code>, <code>latitude</code>, <code>longitude</code>, and <code>coordinates</code>, providing the necessary data to identify and locate matching drivers.</p>
</li>
<li><p><strong>Output</strong>: The <code>scan</code> query with <code>resultFormat: "compactedList"</code> returns up to 1000 results, which the system can use to select the closest driver.</p>
</li>
</ul>
<p>Druid’s spatial indexes on the <code>coordinates</code> dimension ensure these proximity searches are blazing fast, even over large datasets.</p>
<h2 id="heading-why-apache-druid-for-geospatial-queries">Why Apache Druid for Geospatial Queries?</h2>
<p>Geospatial queries in a ride-sharing platform demand low latency and high throughput, and Druid delivers on both fronts. Here’s why it’s a great choice:</p>
<ul>
<li><p><strong>Sub-Second Query Performance</strong>: Druid’s columnar storage and spatial indexing optimize geospatial queries, enabling sub-second response times for proximity searches.</p>
</li>
<li><p><strong>Real-Time Ingestion</strong>: Druid supports streaming ingestion, so driver locations are available for querying as soon as they’re reported.</p>
</li>
<li><p><strong>Scalability</strong>: Whether handling thousands or millions of location updates per second, Druid scales horizontally to meet demand.</p>
</li>
<li><p><strong>Flexible Query Interface</strong>: Druid’s native query language, with spatial filters, supports complex geospatial queries, complementing its SQL capabilities.</p>
</li>
</ul>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Apache Druid’s geospatial query capabilities empower businesses to unlock insights from location-based data at scale. In the ride-sharing example, we saw how Druid can ingest location data and identify nearby drivers in real time using a native scan query with a spatial filter, leveraging spatial indexes for blazing-fast performance. Whether you’re building a ride-sharing platform or analyzing location data, Druid’s ability to handle geospatial queries makes it a powerful tool for real-time analytics.</p>
<p>Have you tried geospatial queries in Druid? Share your use cases or questions in the comments below!</p>
<h2 id="heading-other-resources">Other Resources</h2>
<ul>
<li><a target="_blank" href="https://druid.apache.org/docs/latest/querying/geo/">https://druid.apache.org/docs/latest/querying/geo/</a></li>
</ul>
]]></content:encoded></item><item><title><![CDATA[Automating Google Cloud with GitHub Actions using gcloud CLI]]></title><description><![CDATA[In today's DevOps landscape, automating tasks and deployments is essential for efficiency and reliability. One powerful way to achieve this is by…]]></description><link>https://nikhilrao.blog/automating-google-cloud-with-github-actions-using-gcloud-cli</link><guid isPermaLink="true">https://nikhilrao.blog/automating-google-cloud-with-github-actions-using-gcloud-cli</guid><category><![CDATA[Google Cloud Platform]]></category><category><![CDATA[GitHub Actions]]></category><category><![CDATA[GitHub]]></category><category><![CDATA[gcloud]]></category><category><![CDATA[gcloud-cli]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Wed, 22 Jan 2025 10:18:24 GMT</pubDate><content:encoded><![CDATA[<p>In today's DevOps landscape, automating tasks and deployments is essential for efficiency and reliability. One powerful way to achieve this is by leveraging GitHub Actions to connect with Google Cloud Platform (GCP) using the <code>gcloud</code> CLI. In this post, I'll walk you through setting up a GitHub Action workflow to use the <code>gcloud</code> CLI, allowing you to perform various tasks in GCP directly from your GitHub repository.</p>
<h1 id="heading-prerequisites">Prerequisites</h1>
<ol>
<li><p><strong>Google Cloud Account</strong>: You need access to GCP and sufficient permissions to perform actions, such as deploying applications or managing resources.</p>
</li>
<li><p><strong>GitHub Repository</strong>: Ensure you have a repository where you want to set up the workflow.</p>
</li>
<li><p><strong>gcloud CLI</strong>: Familiarity with the <code>gcloud</code> CLI is helpful, but not required, as we'll cover the basic commands needed.</p>
</li>
</ol>
<h1 id="heading-step-1-set-up-a-service-account-in-gcp">Step 1: Set Up a Service Account in GCP</h1>
<p>First, create a service account in your Google Cloud project with the necessary permissions:</p>
<ol>
<li><p>Navigate to the <strong>IAM &amp; Admin</strong> section in the Google Cloud Console.</p>
</li>
<li><p>Select <strong>Service Accounts</strong> and create a new service account.</p>
</li>
<li><p>Assign the necessary roles for the actions you wish to automate. For example, you might need roles such as <code>roles/storage.admin</code> for Cloud Storage tasks or <code>roles/compute.admin</code> for Compute Engine tasks.</p>
</li>
<li><p>Generate a JSON key for the service account and download it. You'll use this key to authenticate in your GitHub Actions workflow.</p>
</li>
</ol>
<h1 id="heading-step-2-store-service-account-credentials-in-github-secrets">Step 2: Store Service Account Credentials in GitHub Secrets</h1>
<ol>
<li><p>Go to your GitHub repository.</p>
</li>
<li><p>Navigate to <strong>Settings</strong> &gt; <strong>Secrets and variables</strong> &gt; <strong>Actions</strong>.</p>
</li>
<li><p>Click on <strong>New repository secret</strong>.</p>
</li>
<li><p>Add a new secret with the name <code>GCP_SERVICE_ACCOUNT_KEY</code>.</p>
</li>
<li><p>Copy the content of the JSON key file and paste it as the secret value.</p>
</li>
</ol>
<h1 id="heading-step-3-create-your-github-action-workflow">Step 3: Create Your GitHub Action Workflow</h1>
<p>Create a GitHub Actions workflow file in your repository. Here’s an example configuration:</p>
<pre><code class="lang-yaml"><span class="hljs-attr">name:</span> <span class="hljs-string">GCP</span> <span class="hljs-string">Deployment</span>

<span class="hljs-attr">on:</span>
  <span class="hljs-attr">push:</span>
    <span class="hljs-attr">branches:</span>
      <span class="hljs-bullet">-</span> <span class="hljs-string">main</span>

<span class="hljs-attr">jobs:</span>
  <span class="hljs-attr">deploy:</span>
    <span class="hljs-attr">runs-on:</span> <span class="hljs-string">ubuntu-latest</span>

    <span class="hljs-attr">steps:</span>
    <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Checkout</span> <span class="hljs-string">repository</span>
      <span class="hljs-attr">uses:</span> <span class="hljs-string">actions/checkout@v3</span>

    <span class="hljs-bullet">-</span> <span class="hljs-attr">id:</span> <span class="hljs-string">'auth'</span>
      <span class="hljs-attr">uses:</span> <span class="hljs-string">'google-github-actions/auth@v2'</span>
      <span class="hljs-attr">with:</span>
        <span class="hljs-attr">credentials_json:</span> <span class="hljs-string">'$<span class="hljs-template-variable">{{ secrets.GCP_SERVICE_ACCOUNT_KEY }}</span>'</span>

    <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">'Set up Cloud SDK'</span>
      <span class="hljs-attr">uses:</span> <span class="hljs-string">'google-github-actions/setup-gcloud@v2'</span>

    <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">'Use gcloud CLI'</span>
      <span class="hljs-attr">run:</span> <span class="hljs-string">'gcloud info'</span>
</code></pre>
<h3 id="heading-explanation">Explanation</h3>
<ul>
<li>The <code>auth</code> action sets <a target="_blank" href="https://cloud.google.com/docs/authentication/application-default-credentials">Application Default Credentials</a><a target="_blank" href="https://cloud.google.com/docs/authentication/application-default-credentials">, then the <code>setup-gcloud</code> action</a> references these credentials to configure <a target="_blank" href="https://cloud.google.com/docs/authentication/gcloud#gcloud-credentials">gcloud credentials</a><a target="_blank" href="https://cloud.google.com/docs/authentication/gcloud#gcloud-credentials">.</a> The <code>setup-gcloud</code> action installs the Cloud SDK (<code>gcloud</code>).</li>
</ul>
<h1 id="heading-step-4-test-and-validate">Step 4: Test and Validate</h1>
<p>Commit and push your changes to your GitHub repository. Navigate to the <strong>Actions</strong> tab to see the workflow run and validate the output. Make sure that all steps are completed successfully and that the desired actions are performed in your GCP environment.</p>
]]></content:encoded></item><item><title><![CDATA[How to Run a Scheduled Bash Script with Google Cloud Platform]]></title><description><![CDATA[Why would you want to run a scheduled Bash script in the cloud? Couldn’t you just run cron on your local machine? Running a bash script in the cloud rather…]]></description><link>https://nikhilrao.blog/how-to-run-a-scheduled-bash-script-with-google-cloud-platform</link><guid isPermaLink="true">https://nikhilrao.blog/how-to-run-a-scheduled-bash-script-with-google-cloud-platform</guid><category><![CDATA[Google Cloud Platform]]></category><category><![CDATA[google cloud run]]></category><category><![CDATA[Docker]]></category><category><![CDATA[google-cloud-scheduler]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Wed, 22 Jan 2025 06:06:14 GMT</pubDate><content:encoded><![CDATA[<hr />
<h1 id="heading-overview">Overview</h1>
<p>Why would you want to run a scheduled Bash script in the cloud?</p>
<p>Couldn’t you just run cron on your local machine?</p>
<p>Running a bash script in the cloud rather than on a local machine offers developers a multitude of advantages that can enhance both</p>
<ol>
<li><p>Efficiency — developers can ensure greater reliability and availability, as cloud services often offer redundancy and failover mechanisms that protect against data loss and downtime</p>
</li>
<li><p>Scalability — cloud environments can provide enhanced computational power and resources that can handle more complex or resource-intensive scripts, reducing execution time and improving performance.</p>
</li>
</ol>
<p>To run bash scripts on a schedule in Google Cloud, you will need to setup the following components (detailed in the next sections):</p>
<ol>
<li><p><strong>Docker</strong>— The script with all of the dependencies. We will be packaging the Bash script and all required dependencies with a <a target="_blank" href="https://docs.docker.com/get-started/docker-concepts/building-images/writing-a-dockerfile/">Dockerfile</a>.</p>
</li>
<li><p><strong>Cloud Build and Artifact Registry—</strong> Build the created Dockerfile into a Docker Image and push to Artifact Registry.</p>
</li>
<li><p><strong>Cloud Run Jobs</strong> — Compute to execute the script</p>
</li>
<li><p><strong>Cloud Scheduler</strong> — A service to trigger the bash script to run on a specified schedule.</p>
</li>
</ol>
<hr />
<h1 id="heading-docker">Docker</h1>
<p>The bash script and any dependencies it uses must be packaged into a dockerfile and then a docker container needs to be created as an artifact to be used in Cloud Run.</p>
<p>Lets first create a simple “hello world” bash script and save it as <em>hello_</em><a target="_blank" href="http://world.sh"><em>world.sh</em></a></p>
<pre><code class="lang-plaintext">#!/bin/bash
echo "Hello, world! The time is $(date)."
</code></pre>
<p>Next, lets create a Dockerfile to run the bash script when the container is created. Save the filename as <em>Dockerfile</em> in the same directory as the bash script.</p>
<pre><code class="lang-plaintext"># Use a minimal base image with bash
FROM ubuntu:latest

# Set the working directory inside the container
WORKDIR /app

# Copy the bash script into the container
COPY hello_world.sh .

# Make the script executable
RUN chmod +x hello_world.sh

# Set the default command to execute the script
CMD ["./hello_world.sh"]
</code></pre>
<hr />
<h1 id="heading-cloud-build-and-artifact-registry">Cloud Build and Artifact Registry</h1>
<p>Create a Docker repository in Artifact Registry</p>
<pre><code class="lang-plaintext">gcloud artifacts repositories create &lt;YOUR_REPO_NAME&gt; --repository-format=docker \
    --location=us-west2 --description="Docker repository"
</code></pre>
<p>Build the docker image and push to Artifact Registry with Cloud Build</p>
<pre><code class="lang-plaintext">gcloud builds submit --region=us-west2 --tag us-west2-docker.pkg.dev/&lt;PROJECT_ID&gt;/&lt;YOUR_REPO_NAME&gt;/hello-world-image
</code></pre>
<p>If the docker image was built and pushed correctly, it should be in the output of this command</p>
<pre><code class="lang-plaintext">gcloud artifacts docker images list us-west2-docker.pkg.dev/&lt;PROJECT_ID&gt;/&lt;YOUR_REPO_NAME&gt;/
</code></pre>
<hr />
<h1 id="heading-cloud-run-job">Cloud Run Job</h1>
<p>To actually run our bash script contained in the Docker Image we created in the previous section, we need to deploy the image as a Cloud Run Job.</p>
<p>Unlike a Cloud Run service, which listens for and serves requests, a Cloud Run job only runs its tasks and exits when finished. A job does not listen for or serve requests.</p>
<p>Run the following gcloud command to deploy a container and create a Cloud Run job.</p>
<pre><code class="lang-plaintext">gcloud run jobs create my-script-job --image=us-west2-docker.pkg.dev/&lt;PROJECT_ID&gt;/&lt;YOUR_REPO_NAME&gt;/hello-world-image:tag1 --region us-west2
</code></pre>
<p>If done correctly, you should be able to execute the job and see the results with the command</p>
<pre><code class="lang-plaintext">gcloud run jobs execute my-script-job --region=us-west2
</code></pre>
<pre><code class="lang-plaintext">gcloud run jobs executions describe &lt;execution_id&gt; --region=us-west2
</code></pre>
<hr />
<h1 id="heading-cloud-scheduler">Cloud Scheduler</h1>
<p>For the final step, we need to trigger our script to run on a schedule. We will be using <a target="_blank" href="https://cloud.google.com/scheduler/docs/schedule-run-cron-job">Google Cloud Scheduler, which allows you to schedule and ru</a>n a cron job to trigger HTTP/S endpoints and Cloud Run jobs.</p>
<p>To create a Cloud Scheduler Job to trigger the Cloud Run job we created in the previous section, run the following gcloud comm<a target="_blank" href="https://cloud.google.com/scheduler/docs/schedule-run-cron-job">and.</a></p>
<pre><code class="lang-plaintext">gcloud scheduler jobs create http my-scheduler-job-1 \
  --location us-west2 \
  --schedule="0 * * * *" \
  --uri="https://us-west2-run.googleapis.com/apis/run.googleapis.com/v1/projects/PROJECT-ID/jobs/my-script-job:run" \
  --http-method POST \
  --location=us-west2
</code></pre>
<p>The example cron schedule for this example is set to hourly at minute 0. Also, because the <em>oauth-service-account-email</em> argumen<a target="_blank" href="https://cloud.google.com/scheduler/docs/schedule-run-cron-job">t was not explicitly s</a>pecified in the command, it will use the default compute account for authentication to call the Cloud Run jobs API.</p>
<hr />
<h1 id="heading-conclusion">Conclusion</h1>
<p>In conclusion, running a bash script in the cloud rather than on a local machine offers developers a multitude of advantages that can significantly enhance both efficiency and productivity.</p>
<p>By leveraging Docker, Cloud Build, Cloud Run, and Cloud Scheduler, developers gain access to unparalleled scalability, enabling them to run scripts on-demand without worrying about local resource limitations.</p>
<p>This scalability is complemented by the inherent flexibility of Google Cloud Platform, allowing developers to seamlessly integrate their scripts with a variety of tools and services, thereby streamlining workflows and reducing operational overhead.</p>
<hr />
<h1 id="heading-references">References</h1>
<p><a target="_blank" href="https://cloud.google.com/run/docs/create-jobs">https://cloud.google.com/run/docs/create-jobs</a></p>
<p><a target="_blank" href="https://cloud.google.com/scheduler/docs/schedule-run-cron-job">https://cloud.google.com/scheduler/docs/schedule-run-cron-job</a></p>
<p><a target="_blank" href="https://cloud.google.com/build/docs">https://cloud.google.com/build/docs</a></p>
]]></content:encoded></item><item><title><![CDATA[Google Dataflow Optimization: Streaming Engine]]></title><description><![CDATA["By default, the Dataflow pipeline runner executes the steps of your streaming pipeline entirely on worker virtual machines, consuming worker CPU, memory…]]></description><link>https://nikhilrao.blog/google-dataflow-optimization-streaming-engine</link><guid isPermaLink="true">https://nikhilrao.blog/google-dataflow-optimization-streaming-engine</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Wed, 13 Sep 2023 18:24:23 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-what-is-streaming-engine">What is Streaming Engine</h2>
<p>"By default, the Dataflow pipeline runner executes the steps of your streaming pipeline entirely on worker virtual machines, consuming worker CPU, memory, and Persistent Disk storage. Dataflow's Streaming Engine moves pipeline execution out of the worker VMs and into the Dataflow service backend. For more information, see Streaming Engine.</p>
<p>The Streaming Engine model has the following benefits:</p>
<ul>
<li><p>Reduced CPU, memory, and Persistent Disk storage resource usage on the worker VMs. Streaming Engine works best with smaller worker machine types (<code>n1-standard-2</code> instead of <code>n1-standard-4</code>). It doesn't require Persistent Disk beyond a small worker boot disk, leading to less resource and quota consumption.</p>
</li>
<li><p>More responsive <a target="_blank" href="https://cloud.google.com/dataflow/docs/horizontal-autoscaling">Horizontal Autoscaling</a> in response to variations in incoming data volume. Streaming Engine offers smoother, more granular scaling of workers.</p>
</li>
</ul>
<p>Most of the reduction in worker resources comes from offloading the work to the Dataflow service. For that reason, there is a <a target="_blank" href="https://cloud.google.com/dataflow/pricing">charge</a> associated with the use of Streaming Engine."</p>
<h2 id="heading-how-it-works">How it Works</h2>
<p>The core of the Streaming Engine is to assign a key to each message being processed and perform all processing in the context of the key. The key allows the Streaming Engine to track the state across related messages. The state is tracked, using the key, in a Key-Value persistent store. When Dataflow transforms modify the messages, these are persisted in the store. Also, by using keys, the messages can be more easily partitioned and load-balanced among workers, as each worker can process a range of the key space.</p>
<p>For more details, please see this <a target="_blank" href="https://medium.com/google-cloud/streaming-engine-execution-model-1eb2eef69a8e">article</a></p>
<h2 id="heading-pricing">Pricing</h2>
<p>For streaming pipelines, the Dataflow Streaming Engine moves streaming shuffle and state processing out of the worker VMs and into the Dataflow service backend</p>
<p>Streaming Engine usage is billed by the volume of streaming data processed, which depends on the following:</p>
<ul>
<li><p>The volume of data ingested into your streaming pipeline</p>
</li>
<li><p>The complexity of the pipeline</p>
</li>
<li><p>The number of pipeline stages with shuffle operation or with <a target="_blank" href="https://beam.apache.org/blog/stateful-processing/">stateful DoFns</a></p>
</li>
</ul>
<p>Examples of what counts as a byte processed include the following items:</p>
<ul>
<li><p>Input flows from data sources</p>
</li>
<li><p>Flows of data from one fused pipeline stage to another fused stage</p>
</li>
<li><p>Flows of data persisted in user-defined state or used for windowing</p>
</li>
<li><p>Output messages to data sinks, such as to Pub/Sub or BigQuery</p>
</li>
</ul>
<p>Please see this <a target="_blank" href="https://cloud.google.com/dataflow/pricing#streaming-data">Google Cloud Platform pricing page</a> for the latest information on costs per region.</p>
<h2 id="heading-how-to-use-streaming-engine">How to Use Streaming Engine</h2>
<p>All you need to do is add the following CLI parameter when you are deploying your Dataflow pipeline. <code>--enable_streaming_engine</code> . Then, your dataflow streaming job will start to use the streaming engine for processing!</p>
<h2 id="heading-useful-links">Useful Links</h2>
<ol>
<li><p><a target="_blank" href="https://cloud.google.com/dataflow/docs/streaming-engine">https://cloud.google.com/dataflow/docs/streaming-engine</a></p>
</li>
<li><p><a target="_blank" href="https://medium.com/google-cloud/streaming-engine-execution-model-1eb2eef69a8e">https://medium.com/google-cloud/streaming-engine-execution-model-1eb2eef69a8e</a></p>
</li>
</ol>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: Windowing]]></title><description><![CDATA[Windows are a way to group your data by their event times. But, why do you want to group on time? So you can apply aggregations! An example might be if you…]]></description><link>https://nikhilrao.blog/apache-beam-windowing</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-windowing</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><category><![CDATA[windowing]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Tue, 05 Sep 2023 03:53:09 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-what-are-windows">What are Windows?</h2>
<p>Windows are a way to group your data by their event times. But, why do you want to group on time?</p>
<p>So you can apply aggregations!</p>
<p>An example might be if you have a stream of analytic data coming from mobile phones and want to count the number of times someone opens your app <strong>per hour</strong>. In this scenario, you want to aggregate data within some time bounds.</p>
<h2 id="heading-bounded-vs-unbounded-collections">Bounded vs. Unbounded Collections</h2>
<p>Typically, in other data processing frameworks, you only run an ETL pipeline on batches of data (possibly from a filesystem or database). However, when using Apache Beam, you can run your pipeline on <strong>BOTH</strong> streaming and batch data with the same code.</p>
<p>Bounded data is data that comes from a fixed source like a file. Unbounded data comes from a continuously updating source like a subscription. Windowing is a mechanism to turn your unbounded data stream into a bounded data set which can then be aggregated on.</p>
<h2 id="heading-how-to-use-windowing-functions">How to use Windowing Functions</h2>
<p>You should use the windowing functions on a PCollection before you apply a grouping transform, like <code>GroupByKey</code> or <code>Combine</code>. These transforms will then group on both the timestamp (using the window) and the key (if one is present).</p>
<h2 id="heading-built-in-window-functions">Built-In Window Functions</h2>
<p>There are 4 window functions Apache Beam provides:</p>
<ol>
<li><p>Fixed Time Windows - Non-overlapping time intervals with consistent duration</p>
</li>
<li><p>Sliding Time Windows - Overlapping time intervals with a <code>period</code> and <code>window duration</code></p>
</li>
<li><p>Session Windows - windows within a certain gap duration of the next element. These windows also apply on a per-key basis</p>
</li>
<li><p>Single Global Window - the default window which contains all data and no late data</p>
</li>
</ol>
<h2 id="heading-example-counting-words-with-fixed-minute-windows">Example: Counting Words with Fixed Minute Windows</h2>
<pre><code class="lang-java">    <span class="hljs-comment">// Create some input data with timestamps</span>
    List&lt;String&gt; inputData = Arrays.asList(<span class="hljs-string">"foo"</span>, <span class="hljs-string">"bar"</span>, <span class="hljs-string">"foo"</span>, <span class="hljs-string">"foo"</span>);
    List&lt;Long&gt; timestamps =
        Arrays.asList(
            Duration.standardSeconds(<span class="hljs-number">15</span>).getMillis(),
            Duration.standardSeconds(<span class="hljs-number">30</span>).getMillis(),
            Duration.standardSeconds(<span class="hljs-number">45</span>).getMillis(),
            Duration.standardSeconds(<span class="hljs-number">90</span>).getMillis());

    <span class="hljs-comment">// Create a PCollection from the input data with timestamps</span>
    PCollection&lt;String&gt; items = pipeline.apply(Create.timestamped(inputData, timestamps));

    <span class="hljs-comment">// Create a windowed PCollection</span>
    PCollection&lt;String&gt; windowedItems =
        items.apply(Window.into(FixedWindows.of(Duration.standardMinutes(<span class="hljs-number">1</span>))));

    PCollection&lt;KV&lt;String, Long&gt;&gt; windowedCounts = windowedItems.apply(Count.perElement());
</code></pre>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: Values Transform]]></title><description><![CDATA[What if you only care about the values of your PCollection and not necessarily the keys? Maybe you have KV pairs for words and their counts in a string…]]></description><link>https://nikhilrao.blog/apache-beam-values-transform</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-values-transform</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Mon, 04 Sep 2023 02:20:37 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>What if you only care about the values of your PCollection and not necessarily the keys? Maybe you have KV pairs for words and their counts in a string (KV&lt;String, Integer&gt;) and want to use the counts in another PCollection.</p>
<p>You should use the <code>Values</code> transform!</p>
<h2 id="heading-when-you-should-use-the-values-transform">When You Should Use the Values Transform</h2>
<p>When you only want the values in from a KV PCollection.</p>
<h2 id="heading-how-to-use-the-keys-transform">How to Use the Keys Transform</h2>
<p>Just apply the built-in Transform to a PCollection of KVs. The output PCollection will have the type of the values in the KV and be flattened for all input elements.</p>
<h2 id="heading-example-extract-values-from-word-counts">Example: Extract Values from Word Counts</h2>
<pre><code class="lang-java">    <span class="hljs-comment">// Create key/value pairs</span>
    PCollection&lt;KV&lt;String, Integer&gt;&gt; pairs =
        pipeline.apply(
            Create.of(KV.of(<span class="hljs-string">"one"</span>, <span class="hljs-number">1</span>), KV.of(<span class="hljs-string">"two"</span>, <span class="hljs-number">2</span>), KV.of(<span class="hljs-string">"three"</span>, <span class="hljs-number">3</span>), KV.of(<span class="hljs-string">"four"</span>, <span class="hljs-number">4</span>)));
    <span class="hljs-comment">// Returns only the values of the collection: PCollection&lt;KV&lt;K,V&gt;&gt; -&gt;</span>
    <span class="hljs-comment">// PCollection&lt;V&gt;</span>
    PCollection&lt;Integer&gt; valuesOnly = pairs.apply(Values.create());

<span class="hljs-comment">// results in</span>
<span class="hljs-comment">// 3</span>
<span class="hljs-comment">// 1</span>
<span class="hljs-comment">// ... etc.</span>
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: WithKeys Transform]]></title><description><![CDATA[If you need to convert a PCollection into a KV pair using dynamically generated keys derived from the input element, you should check out the WithKeys…]]></description><link>https://nikhilrao.blog/apache-beam-withkeys-transform</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-withkeys-transform</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Mon, 04 Sep 2023 02:14:22 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>If you need to convert a PCollection into a KV pair using dynamically generated keys derived from the input element, you should check out the <code>WithKeys</code> transform.</p>
<h2 id="heading-when-you-should-use-the-withkeys-transform">When You Should Use the WithKeys Transform</h2>
<p>When you only want to convert a <code>PCollection&lt;V&gt;</code> to a <code>PCollection&lt;KV&lt;K, V&gt;&gt;</code>. You can then use aggregations or transforms your newly generated KV's.</p>
<h2 id="heading-how-to-use-the-withkeys-transform">How to Use the WithKeys Transform</h2>
<p>Just apply the built-in Transform to any PCollection and supply a function which will take the input element as an argument and return a value which will be used as a key.</p>
<h2 id="heading-example-create-a-kv-pair-from-a-list-of-words">Example: Create a KV pair from a list of Words</h2>
<pre><code class="lang-java">    PCollection&lt;String&gt; words = pipeline.apply(Create.of(<span class="hljs-string">"Hello"</span>, <span class="hljs-string">"World"</span>, <span class="hljs-string">"Beam"</span>, <span class="hljs-string">"is"</span>, <span class="hljs-string">"fun"</span>));
    PCollection&lt;KV&lt;Integer, String&gt;&gt; lengthAndWord =
        words.apply(
            WithKeys.of(
                <span class="hljs-keyword">new</span> SerializableFunction&lt;String, Integer&gt;() {
                  <span class="hljs-meta">@Override</span>
                  <span class="hljs-function"><span class="hljs-keyword">public</span> Integer <span class="hljs-title">apply</span><span class="hljs-params">(String s)</span> </span>{
                    <span class="hljs-keyword">return</span> s.length();
                  }
                }));
<span class="hljs-comment">// returns</span>
<span class="hljs-comment">// KV{5, World}</span>
<span class="hljs-comment">// KV{5, Hello}</span>
<span class="hljs-comment">// ... etc</span>
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: ToString Transform]]></title><description><![CDATA[If you want to convert every input element from a PCollection to a string, you should check out the ToString transforms. It can do everything from simply…]]></description><link>https://nikhilrao.blog/apache-beam-tostring-transform</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-tostring-transform</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Mon, 04 Sep 2023 02:07:36 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>If you want to convert every input element from a PCollection to a string, you should check out the <code>ToString</code> transforms. It can do everything from simply converting an object to a string by implicitly calling its <code>toString()</code> method to concatenate a kv pair with a custom delimiter.</p>
<h2 id="heading-when-you-should-use-the-tostring-transform">When You Should Use the ToString Transform</h2>
<p>You should use the <code>ToString</code> transforms when you want to perform the following transformations on your input data:</p>
<ol>
<li><p>Transform each element into a string using the <code>Object.toString()</code> method</p>
</li>
<li><p>Transform an input element of type <code>Iterables</code> into a string using a delimiter</p>
</li>
<li><p>Transform an input element of type <code>KV</code> into a string using a delimiter</p>
</li>
</ol>
<h2 id="heading-how-to-use-the-tostring-transform">How to Use the ToString Transform</h2>
<p>Just apply the built-in transform to a PCollection of elements, lists, or KVs. The output PCollection will always have the type of the <code>String</code>.</p>
<h2 id="heading-example-convert-kv-to-string">Example: Convert KV to String</h2>
<pre><code class="lang-java">    <span class="hljs-comment">// Create key-value pairs</span>
    PCollection&lt;KV&lt;String, String&gt;&gt; pairs =
        pipeline.apply(
            Create.of(
                KV.of(<span class="hljs-string">"fall"</span>, <span class="hljs-string">"apple"</span>),
                KV.of(<span class="hljs-string">"spring"</span>, <span class="hljs-string">"strawberry"</span>),
                KV.of(<span class="hljs-string">"winter"</span>, <span class="hljs-string">"orange"</span>),
                KV.of(<span class="hljs-string">"summer"</span>, <span class="hljs-string">"peach"</span>),
                KV.of(<span class="hljs-string">"spring"</span>, <span class="hljs-string">"cherry"</span>),
                KV.of(<span class="hljs-string">"fall"</span>, <span class="hljs-string">"pear"</span>)));
    <span class="hljs-comment">// Use ToString on key-value pairs</span>
    PCollection&lt;String&gt; result = pairs.apply(ToString.kvs());

    <span class="hljs-comment">// results in a PCollection containing</span>
    <span class="hljs-comment">// fall,apple</span>
    <span class="hljs-comment">// string,strawberry</span>
    <span class="hljs-comment">// ...etc</span>
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: Regex]]></title><description><![CDATA[Do you have a custom DoFn that applies a regular expression (regex) pattern to the input element? Well, did you know there is a built-in Apache Beam…]]></description><link>https://nikhilrao.blog/apache-beam-regex</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-regex</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Mon, 04 Sep 2023 01:39:49 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>Do you have a custom DoFn that applies a regular expression (regex) pattern to the input element? Well, did you know there is a built-in Apache Beam transform called <code>Regex</code> which can simplify your code?</p>
<h2 id="heading-when-you-should-use-the-keys-transform">When You Should Use the Keys Transform</h2>
<p>You should use the <code>Regex</code> transform if you want to perform the following for every element in a PCollection:</p>
<ol>
<li><p>Filter for strings matching a certain pattern</p>
</li>
<li><p>Replace a pattern of a string with another string</p>
</li>
<li><p>Split a string with a specific delimiter</p>
</li>
</ol>
<h2 id="heading-how-to-use-the-keys-transform">How to Use the Keys Transform</h2>
<p>Just apply the built-in transform to a PCollection of strings. Use <code>Regex.Matches</code> to filter input elements based on a certain regex pattern. Use <code>Regex.ReplaceAll</code> to replace substrings in a string. Use <code>Regex.Split</code> to split a string into multiple strings using a delimiter.</p>
<h2 id="heading-example-filter-strings-with-email-regex-pattern">Example: Filter Strings with Email Regex Pattern</h2>
<pre><code class="lang-java"><span class="hljs-comment">// Create a collection with strings</span>
    PCollection&lt;String&gt; emails =
        pipeline.apply(
            Create.of(
                <span class="hljs-string">"johndoe@gmail.com"</span>,
                <span class="hljs-string">"sarahsmith@yahoo.com"</span>,
                <span class="hljs-string">"mikebrown@outlook.com"</span>,
                <span class="hljs-string">"amandajohnson"</span>,
                <span class="hljs-string">"davidlee"</span>,
                <span class="hljs-string">"emilyrodriguez"</span>));

    <span class="hljs-comment">// Take only strings which match the email regex</span>
    PCollection&lt;String&gt; result =
        emails.apply(Regex.matches(<span class="hljs-string">"([a-zA-Z0-9._%-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,6})"</span>));
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: Partition Transform]]></title><description><![CDATA[If you want to split a PCollection into multiple PCollections based on a function, use the Partition transform. When you want to split the input…]]></description><link>https://nikhilrao.blog/apache-beam-partition-transform</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-partition-transform</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Mon, 04 Sep 2023 00:46:42 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>If you want to split a PCollection into multiple PCollections based on a function, use the <code>Partition</code> transform.</p>
<h2 id="heading-when-you-should-use-the-keys-transform">When You Should Use the Keys Transform</h2>
<p>When you want to split the input PCollection into multiple PCollections. You might want to partition a collection of strings based on their length. Then, truncate the longer strings into smaller sizes, but keep the shorter strings the same.</p>
<h2 id="heading-how-to-use-the-keys-transform">How to Use the Keys Transform</h2>
<p>Just apply the builtin Transform to a PCollection with the following arguments:</p>
<ol>
<li><p>Number of Partitions</p>
</li>
<li><p>Partition function - This function must return an integer between 0 and the number of partitions</p>
</li>
</ol>
<h2 id="heading-example-partition-based-on-percentile">Example: Partition based on Percentile</h2>
<pre><code class="lang-java">    <span class="hljs-comment">// Provide an int value with the desired number of result partitions, and a</span>
    <span class="hljs-comment">// PartitionFn that represents the</span>
    <span class="hljs-comment">// partitioning function. In this example, we define the PartitionFn in-line.</span>
    <span class="hljs-comment">// Returns a PCollectionList</span>
    <span class="hljs-comment">// containing each of the resulting partitions as individual PCollection</span>
    <span class="hljs-comment">// objects.</span>
    PCollection&lt;Student&gt; students =
        pipeline.apply(
            Create.of(
                Student.of(<span class="hljs-string">"Amy"</span>, <span class="hljs-number">88</span>),
                Student.of(<span class="hljs-string">"Bob"</span>, <span class="hljs-number">87</span>),
                Student.of(<span class="hljs-string">"Chris"</span>, <span class="hljs-number">49</span>),
                Student.of(<span class="hljs-string">"Dylan"</span>, <span class="hljs-number">62</span>),
                Student.of(<span class="hljs-string">"Ellen"</span>, <span class="hljs-number">78</span>),
                Student.of(<span class="hljs-string">"Francis"</span>, <span class="hljs-number">53</span>)));
    <span class="hljs-comment">// Split students up into 10 partitions, by percentile:</span>
    PCollectionList&lt;Student&gt; studentsByPercentile =
        students.apply(
            Partition.of(
                <span class="hljs-number">10</span>,
                <span class="hljs-keyword">new</span> Partition.PartitionFn&lt;Student&gt;() {
                  <span class="hljs-meta">@Override</span>
                  <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">int</span> <span class="hljs-title">partitionFor</span><span class="hljs-params">(Student student, <span class="hljs-keyword">int</span> numPartitions)</span> </span>{
                    <span class="hljs-keyword">return</span> student.getPercentile() <span class="hljs-comment">// 0..99</span>
                        * numPartitions
                        / <span class="hljs-number">100</span>;
                  }
                }));

    <span class="hljs-comment">// You can extract each partition from the PCollectionList using the get method,</span>
    <span class="hljs-comment">// as follows:</span>
    PCollection&lt;Student&gt; fortiethPercentile = studentsByPercentile.get(<span class="hljs-number">4</span>);
    fortiethPercentile.apply(ParDo.of(<span class="hljs-keyword">new</span> LogOutput&lt;&gt;(<span class="hljs-string">"Fortieth percentile: "</span>)));
    pipeline.run();
  }
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: ParDo]]></title><description><![CDATA[The General Purpose Transform]]></description><link>https://nikhilrao.blog/apache-beam-pardo</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-pardo</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Sun, 03 Sep 2023 23:43:32 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>If you want to apply a function which transforms each element in an input collection, you should use the <code>ParDo</code> transform.</p>
<h2 id="heading-when-you-should-use-the-pardo-transform">When You Should Use the ParDo Transform</h2>
<p>Any time you want to perform some generic processing function on each element of a PCollection.</p>
<h2 id="heading-how-to-use-the-pardo-transform">How to Use the ParDo Transform</h2>
<p>To perform processing on an element using the <code>ParDo</code> transform, all you need is a Java class which is a subclass of the <code>DoFn</code> abstract class which contains your transformation logic. Your subclass must override the <code>@processElement</code> function with the transformation code (see below examples).</p>
<h2 id="heading-example-filter-word-counts-with-max-length">Example: Filter Word Counts with Max Length</h2>
<pre><code class="lang-java">  <span class="hljs-comment">// The input PCollection to ParDo.</span>
  PCollection&lt;String&gt; words = ...;

  <span class="hljs-comment">// Apply a ParDo that takes maxWordLengthCutOffView as a side input.</span>
  PCollection&lt;String&gt; wordsBelowCutOff =
  words.apply(ParDo
      .of(<span class="hljs-keyword">new</span> DoFn&lt;String, String&gt;() {
          <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">processElement</span><span class="hljs-params">(ProcessContext c)</span> </span>{
            String word = c.element();
            <span class="hljs-keyword">int</span> maxLength = <span class="hljs-number">10</span>;
            <span class="hljs-keyword">if</span> (word.length() &lt;= maxLength) {
              c.output(word);
            }
          }
      })
  );
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: KvSwap]]></title><description><![CDATA[Swap the Keys and Values of a Key-Value Pair]]></description><link>https://nikhilrao.blog/apache-beam-kvswap</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-kvswap</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Sun, 03 Sep 2023 22:20:46 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>What if you have a Key-Value pair, but want to group on the values, not the keys? Should you write a custom doFn to switch the keys and values?</p>
<p>NO!</p>
<p>You should use the <code>KvSwap</code> transform!</p>
<h2 id="heading-when-you-should-use-the-kvswap-transform">When You Should Use the KvSwap Transform</h2>
<p>When you want to easily swap the keys and values of a PCollection of KV pairs.</p>
<h2 id="heading-how-to-use-the-kvswap-transform">How to Use the KvSwap Transform</h2>
<p>Just apply the built-in Transform to a PCollection of KVs. The output PCollection will have the Keys and Values swapped from the input KV.</p>
<h2 id="heading-example-swap-keys-and-values-from-word-counts">Example: Swap Keys and Values from Word Counts</h2>
<pre><code class="lang-java">    <span class="hljs-comment">// Create key/value pairs</span>
    PCollection&lt;KV&lt;String, Integer&gt;&gt; pairs =
        pipeline.apply(
            Create.of(KV.of(<span class="hljs-string">"one"</span>, <span class="hljs-number">1</span>), KV.of(<span class="hljs-string">"two"</span>, <span class="hljs-number">2</span>), KV.of(<span class="hljs-string">"three"</span>, <span class="hljs-number">3</span>), KV.of(<span class="hljs-string">"four"</span>, <span class="hljs-number">4</span>)));
    <span class="hljs-comment">// Returns KV collection with keys and values swapped: PCollection&lt;KV&lt;K,V&gt;&gt; -&gt;</span>
    <span class="hljs-comment">// PCollection&lt;KV&lt;V,K&gt;&gt;</span>
    PCollection&lt;KV&lt;Integer, String&gt;&gt; swap = pairs.apply(KvSwap.create());
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: Keys]]></title><description><![CDATA[Get the Keys from a Key-Value Pair]]></description><link>https://nikhilrao.blog/apache-beam-keys</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-keys</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><category><![CDATA[Google Cloud Platform]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Sun, 03 Sep 2023 21:49:52 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>What if you only care about the keys of your PCollection and not necessarily the values? Maybe you have KV pairs for words and their counts in a string (KV&lt;String, Integer&gt;) and want to use the list of words in another PCollection.</p>
<p>You should use the <code>Keys</code> transform!</p>
<h2 id="heading-when-you-should-use-the-keys-transform">When You Should Use the Keys Transform</h2>
<p>When you only want the keys in from a KV PCollection.</p>
<h2 id="heading-how-to-use-the-keys-transform">How to Use the Keys Transform</h2>
<p>Just apply the builtin Transform to a PCollection of KVs. The output PCollection will have the type of the Keys in the KV and be flattened for all input elements.</p>
<h2 id="heading-example-extract-keys-from-word-counts">Example: Extract Keys from Word Counts</h2>
<pre><code class="lang-java">    <span class="hljs-comment">// Create key/value pairs</span>
    PCollection&lt;KV&lt;String, Integer&gt;&gt; pairs =
        pipeline.apply(
            Create.of(KV.of(<span class="hljs-string">"one"</span>, <span class="hljs-number">1</span>), KV.of(<span class="hljs-string">"two"</span>, <span class="hljs-number">2</span>), KV.of(<span class="hljs-string">"three"</span>, <span class="hljs-number">3</span>), KV.of(<span class="hljs-string">"four"</span>, <span class="hljs-number">4</span>)));
    <span class="hljs-comment">// Returns only the values of the collection: PCollection&lt;KV&lt;K,V&gt;&gt; -&gt;</span>
    <span class="hljs-comment">// PCollection&lt;V&gt;</span>
    PCollection&lt;String&gt; valuesOnly = pairs.apply(Keys.create());
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: Filter]]></title><description><![CDATA[Use Apache Beam's built-in Filter Transform to Simplify your Pipelines]]></description><link>https://nikhilrao.blog/apache-beam-filter</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-filter</guid><category><![CDATA[google cloud]]></category><category><![CDATA[Apache Beam]]></category><category><![CDATA[Java]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Sun, 03 Sep 2023 21:13:37 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>Have you ever wanted to filter data from a PCollection?</p>
<p>With the <code>Filter</code> transform, you can!</p>
<p>Given a function that takes an input element and returns <strong>True</strong> or <strong>False</strong>, the Filter transform will only allow elements that return True to proceed through the pipeline and to the next transform. Also, if the elements of the input PCollection are Java Objects that are <code>Comparable</code>, you can use built-in Filters which will filter out elements based on the elements' natural ordering (see below for examples).</p>
<h2 id="heading-when-you-should-use-the-filter-transform">When You Should Use the Filter Transform</h2>
<p>You should use the <code>Filter</code> transform whenever you need to clean a PCollection based on certain criteria.</p>
<h2 id="heading-how-to-use-the-filter-transform">How to Use the Filter Transform</h2>
<p>Using this transform is quite easy. All you need is a function which returns a boolean (which includes lambda functions) for whether you want to keep the input element (True) or discard the element (False). It will then allow all elements which returned True to go to the next transform in the pipeline.</p>
<h2 id="heading-example-filtering-with-a-function">Example: Filtering with a function</h2>
<pre><code class="lang-java">PCollection&lt;String&gt; allStrings = Create.of(<span class="hljs-string">"Hello"</span>, <span class="hljs-string">"world"</span>, <span class="hljs-string">"hi"</span>);
PCollection&lt;String&gt; longStrings = allStrings
    .apply(Filter.by(<span class="hljs-keyword">new</span> SerializableFunction&lt;String, Boolean&gt;() {
      <span class="hljs-meta">@Override</span>
      <span class="hljs-function"><span class="hljs-keyword">public</span> Boolean <span class="hljs-title">apply</span><span class="hljs-params">(String input)</span> </span>{
        <span class="hljs-keyword">return</span> input.length() &gt; <span class="hljs-number">3</span>;
      }
    }));
</code></pre>
<h2 id="heading-example-filtering-with-a-builtin-filter">Example: Filtering with a Builtin Filter</h2>
<pre><code class="lang-java">PCollection&lt;Long&gt; numbers = Create.of(<span class="hljs-number">2L</span>, <span class="hljs-number">3L</span>, <span class="hljs-number">4L</span>, <span class="hljs-number">5L</span>);
PCollection&lt;Long&gt; bigNumbers = numbers.apply(Filter.greaterThan(<span class="hljs-number">3</span>));
PCollection&lt;Long&gt; smallNumbers = numbers.apply(Filter.lessThanEq(<span class="hljs-number">3</span>));
</code></pre>
<h2 id="heading-example-filtering-with-a-lambda">Example: Filtering with a Lambda</h2>
<pre><code class="lang-java">PCollection&lt;Long&gt; numbers = Create.of(<span class="hljs-number">2L</span>, <span class="hljs-number">3L</span>, <span class="hljs-number">4L</span>, <span class="hljs-number">5L</span>);
PCollection&lt;Long&gt; bigNumbers = numbers.apply(Filter.by(number -&gt; number &gt; <span class="hljs-number">3</span>));
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Instead of writing a custom Do Function to filter elements in a PCollection, you should use the <code>Filter</code> Transform.</p>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Apache Beam: MapElements and FlatMapElements Transforms]]></title><description><![CDATA[Use Built-in Apache Beam functions to simplify your pipelines]]></description><link>https://nikhilrao.blog/apache-beam-mapelements-and-flatmapelements-transforms</link><guid isPermaLink="true">https://nikhilrao.blog/apache-beam-mapelements-and-flatmapelements-transforms</guid><category><![CDATA[Apache Beam]]></category><category><![CDATA[google cloud]]></category><category><![CDATA[Google Cloud Platform]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Sun, 03 Sep 2023 00:23:24 GMT</pubDate><content:encoded><![CDATA[<h2 id="heading-overview">Overview</h2>
<p>Have you ever had a transformation so simple you want a one-liner instead of creating a new DoFn class or custom Transform?</p>
<p>You should use the built-in Apache Beam transforms <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/elementwise/mapelements/"><strong>MapElements</strong></a> <strong>and</strong> <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/elementwise/flatmapelements/"><strong>FlatMapElements</strong></a>!</p>
<h2 id="heading-mapelements">MapElements</h2>
<p>This convenient transform applies a simple 1 to 1 function over each element in the PCollection. In other words, for each input element, it produces <em>exactly one</em> output element. You can even use Java's built-in lambda functions for additional simplicity!</p>
<h2 id="heading-flatmapelements">FlatMapElements</h2>
<p>This transform is similar to the <em>MapElements</em> transform, but the function you apply to each input element can output <em>more than one</em> output element. If the function does return multiple outputs for the given single input element, it will add each individual output element to the output PCollection, and will <strong>not</strong> add lists or iterables to the output collection.</p>
<p>For example, if you are trying to apply a function returning the prime factorization of an integer element, the output PCollection will be <strong>the combined list</strong> of all the factors of every input integer element. If you had an input PCollection of <code>[4, 6, 9]</code>, the output PCollection would be <code>[2, 2, 2, 3, 3, 3]</code>. This is why the transform is named FlatMap, it will flatten all elements produced by the function.</p>
<h2 id="heading-example-1-replace-a-character-in-each-string-in-a-pcollection">Example 1: Replace a Character in each string in a PCollection</h2>
<p>Ok, let's say you have a PCollection of strings and you want to remove any <code>"."</code> characters (if they happen to be in the string).</p>
<pre><code class="lang-java"><span class="hljs-comment">// The input PCollection.</span>
PCollection&lt;String&gt; words = ...;

<span class="hljs-comment">// Apply a MapElements with an anonymous lambda function to the PCollection words.</span>
<span class="hljs-comment">// Save the result as the PCollection newWord.</span>
PCollection&lt;String&gt; newWords = words.apply(
  MapElements.into(TypeDescriptors.strings())
             .via((String word) -&gt; word.replace(<span class="hljs-string">"."</span>, <span class="hljs-string">""</span>)));
</code></pre>
<p>The <code>.into</code> specifies the output datatype, in this case <code>strings</code>. The <code>.via</code> specifies the function to apply to each element in the PCollection. By the way, the function does not have to be a lambda function, but can be more general (see <a target="_blank" href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/MapElements.html">MapElements JavaDocs</a>).</p>
<h2 id="heading-example-2-extract-words-from-sentences">Example 2: Extract Words from Sentences</h2>
<p>What if you have a PCollection of full sentences and would like to transform it into a PCollection of individual words in the sentences?</p>
<pre><code class="lang-java">    PCollection&lt;String&gt; lines =
        pipeline.apply(Create.of(<span class="hljs-string">"the quick brown fox"</span>, <span class="hljs-string">"jumps over the lazy"</span>, <span class="hljs-string">"dog"</span>));
    <span class="hljs-comment">// Use FlatMapElements to split the lines in PCollection&lt;String&gt; into individual</span>
    <span class="hljs-comment">// words.</span>
    PCollection&lt;String&gt; words =
        lines.apply(
            FlatMapElements.via(
                <span class="hljs-keyword">new</span> InferableFunction&lt;String, List&lt;String&gt;&gt;() {
                  <span class="hljs-meta">@Override</span>
                  <span class="hljs-function"><span class="hljs-keyword">public</span> List&lt;String&gt; <span class="hljs-title">apply</span><span class="hljs-params">(String line)</span> </span>{
                    <span class="hljs-keyword">return</span> Arrays.asList(line.split(<span class="hljs-string">" "</span>));
                  }
                }));
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>The <strong>MapElements</strong> and <strong>FlatMapElements</strong> built-in Apache Beam transforms can simplify your pipelines and code base by allowing you to use these transforms instead of creating custom DoFunctions to perform simple tasks.</p>
<p>Check out other useful transforms from the <a target="_blank" href="https://beam.apache.org/documentation/transforms/java/overview/">official Apache Beam documentation</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Maximize Profit with Scipy Optimize]]></title><description><![CDATA[The problem is a classic product mix-problem, where the objective is to maximize profit given some production and ingredients constraints, in this case it…]]></description><link>https://nikhilrao.blog/maximize-profit-with-scipy-optimize</link><guid isPermaLink="true">https://nikhilrao.blog/maximize-profit-with-scipy-optimize</guid><category><![CDATA[Python]]></category><category><![CDATA[scipy]]></category><category><![CDATA[optimization]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Sun, 07 Aug 2022 03:03:37 GMT</pubDate><content:encoded><![CDATA[<p>The problem is a classic product mix-problem, where the objective is to maximize profit given some production and ingredients constraints, in this case it is a coffee shop with three coffee types.</p>
<pre><code>x0 <span class="hljs-operator">-</span><span class="hljs-operator">&gt;</span> Regular
x1 <span class="hljs-operator">-</span><span class="hljs-operator">&gt;</span> Latte
x2 <span class="hljs-operator">-</span><span class="hljs-operator">&gt;</span> Mocha
</code></pre><p>Constraints:</p>
<pre><code>x0<span class="hljs-operator">+</span>x1<span class="hljs-operator">+</span>x2 <span class="hljs-operator">&lt;</span><span class="hljs-operator">=</span> <span class="hljs-number">500</span>
x1<span class="hljs-operator">+</span>x2 <span class="hljs-operator">&lt;</span><span class="hljs-operator">=</span> <span class="hljs-number">350</span>
x2 <span class="hljs-operator">&lt;</span><span class="hljs-operator">=</span> <span class="hljs-number">125</span>
</code></pre><p>Objective function</p>
<pre><code><span class="hljs-attribute">Maximize</span> x<span class="hljs-number">0</span>*<span class="hljs-number">1</span>.<span class="hljs-number">5</span> + x<span class="hljs-number">1</span>*<span class="hljs-number">2</span>.<span class="hljs-number">00</span> + x<span class="hljs-number">2</span>*<span class="hljs-number">2</span>.<span class="hljs-number">25</span>
</code></pre><p>The problem lies in both your objective function and the constraints. Since you are using scipy's minimize function, you must minimize the negative of the objective function to find the maximum of the function (It is slightly tricky).</p>
<pre><code><span class="hljs-comment">#profit / cup</span>
<span class="hljs-attribute">Reg</span> = <span class="hljs-number">1</span>.<span class="hljs-number">25</span>
<span class="hljs-attribute">Lat</span> = <span class="hljs-number">2</span>.<span class="hljs-number">00</span>
<span class="hljs-attribute">Moc</span> = <span class="hljs-number">2</span>.<span class="hljs-number">25</span>

<span class="hljs-comment">#objective function to maximize</span>
<span class="hljs-attribute">def</span> objective(x):
    <span class="hljs-attribute">return</span> -<span class="hljs-number">1</span>.<span class="hljs-number">0</span>*(x[<span class="hljs-number">0</span>]*Reg + x[<span class="hljs-number">1</span>]*Lat + x[<span class="hljs-number">2</span>]*Moc)
</code></pre><p>If you look at the scipy documentation, all inequalities are in the form g(x)&gt;=0. So, for example, if you want x2 &lt;= 125, you must multiply by negative 1 (to switch the inequality), then add 125 which gets: g(x) = 125-x2 &gt;= 0. So for the rest of your constraints:</p>
<pre><code><span class="hljs-comment">#constraints</span>
<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">cons_total_production</span>(<span class="hljs-params">x</span>):</span>
    <span class="hljs-keyword">return</span> (<span class="hljs-number">-1.0</span>*sum(x))+<span class="hljs-number">500</span>

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">cons_choc</span>(<span class="hljs-params">x</span>):</span>
    <span class="hljs-keyword">return</span> (<span class="hljs-number">-1.0</span>*x[<span class="hljs-number">2</span>])+<span class="hljs-number">125</span>

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">cons_milk</span>(<span class="hljs-params">x</span>):</span>
    <span class="hljs-keyword">return</span> (<span class="hljs-number">-1.0</span>*sum(x[<span class="hljs-number">1</span>:]))+<span class="hljs-number">350</span>
</code></pre><p>Which will give you the outputs:</p>
<pre><code> <span class="hljs-attr">fun:</span> <span class="hljs-number">-918.7499999999989</span>
 <span class="hljs-attr">jac:</span> <span class="hljs-string">array([-1.25,</span> <span class="hljs-number">-2</span><span class="hljs-string">.</span>  <span class="hljs-string">,</span> <span class="hljs-number">-2.25</span><span class="hljs-string">])</span>
 <span class="hljs-attr">message:</span> <span class="hljs-string">'Optimization terminated successfully.'</span>
 <span class="hljs-attr">nfev:</span> <span class="hljs-number">35</span>
 <span class="hljs-attr">nit:</span> <span class="hljs-number">7</span>
 <span class="hljs-attr">njev:</span> <span class="hljs-number">7</span>
 <span class="hljs-attr">status:</span> <span class="hljs-number">0</span>
 <span class="hljs-attr">success:</span> <span class="hljs-literal">True</span>
 <span class="hljs-attr">x:</span> <span class="hljs-string">array([</span> <span class="hljs-number">150</span><span class="hljs-string">.,</span>  <span class="hljs-number">225</span><span class="hljs-string">.,</span>  <span class="hljs-number">125</span><span class="hljs-string">.])</span>

 <span class="hljs-attr">fun:</span> <span class="hljs-number">-890.62499998809</span>
 <span class="hljs-attr">jac:</span> <span class="hljs-string">array([-1.25,</span> <span class="hljs-number">-2</span><span class="hljs-string">.</span>  <span class="hljs-string">,</span> <span class="hljs-number">-2.25</span><span class="hljs-string">])</span>
 <span class="hljs-attr">message:</span> <span class="hljs-string">'Optimization terminated successfully.'</span>
 <span class="hljs-attr">nfev:</span> <span class="hljs-number">10</span>
 <span class="hljs-attr">nit:</span> <span class="hljs-number">2</span>
 <span class="hljs-attr">njev:</span> <span class="hljs-number">2</span>
 <span class="hljs-attr">status:</span> <span class="hljs-number">0</span>
 <span class="hljs-attr">success:</span> <span class="hljs-literal">True</span>
 <span class="hljs-attr">x:</span> <span class="hljs-string">array([</span> <span class="hljs-number">187.5</span><span class="hljs-string">,</span>  <span class="hljs-number">187.5</span><span class="hljs-string">,</span>  <span class="hljs-number">125</span><span class="hljs-string">.</span> <span class="hljs-string">])</span>
</code></pre><p>Linear programming is so cool!</p>
]]></content:encoded></item><item><title><![CDATA[Rotation of a vector in Python]]></title><description><![CDATA[If you want to rotate a 3D vector around a particular axis, take advantage of matrix transformations instead of element wise (like you have written above)…]]></description><link>https://nikhilrao.blog/rotation-of-a-vector-in-python</link><guid isPermaLink="true">https://nikhilrao.blog/rotation-of-a-vector-in-python</guid><category><![CDATA[Python]]></category><category><![CDATA[numpy]]></category><category><![CDATA[Matplotlib]]></category><category><![CDATA[Matrix]]></category><category><![CDATA[rotation]]></category><dc:creator><![CDATA[Nikhil Rao]]></dc:creator><pubDate>Sat, 06 Aug 2022 22:38:13 GMT</pubDate><content:encoded><![CDATA[<p>If you want to rotate a 3D vector around a particular axis, take advantage of <a target="_blank" href="https://en.wikipedia.org/wiki/Rotation_matrix">matrix transformations</a> instead of element wise (like you have written above). Below is code to rotate a 3-D vector around any axis:</p>
<pre><code><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt
<span class="hljs-keyword">from</span> mpl_toolkits.mplot3d <span class="hljs-keyword">import</span> Axes3D

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">unit_vector</span>(<span class="hljs-params">vector</span>):</span>
    <span class="hljs-string">""" Returns the unit vector of the vector."""</span>
    <span class="hljs-keyword">return</span> vector / np.linalg.norm(vector)

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">angle_between</span>(<span class="hljs-params">v1, v2</span>):</span>
    <span class="hljs-string">"""Finds angle between two vectors"""</span>
    v1_u = unit_vector(v1)
    v2_u = unit_vector(v2)
    <span class="hljs-keyword">return</span> np.arccos(np.clip(np.dot(v1_u, v2_u), <span class="hljs-number">-1.0</span>, <span class="hljs-number">1.0</span>))

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">x_rotation</span>(<span class="hljs-params">vector,theta</span>):</span>
    <span class="hljs-string">"""Rotates 3-D vector around x-axis"""</span>
    R = np.array([[<span class="hljs-number">1</span>,<span class="hljs-number">0</span>,<span class="hljs-number">0</span>],[<span class="hljs-number">0</span>,np.cos(theta),-np.sin(theta)],[<span class="hljs-number">0</span>, np.sin(theta), np.cos(theta)]])
    <span class="hljs-keyword">return</span> np.dot(R,vector)

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">y_rotation</span>(<span class="hljs-params">vector,theta</span>):</span>
    <span class="hljs-string">"""Rotates 3-D vector around y-axis"""</span>
    R = np.array([[np.cos(theta),<span class="hljs-number">0</span>,np.sin(theta)],[<span class="hljs-number">0</span>,<span class="hljs-number">1</span>,<span class="hljs-number">0</span>],[-np.sin(theta), <span class="hljs-number">0</span>, np.cos(theta)]])
    <span class="hljs-keyword">return</span> np.dot(R,vector)

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">z_rotation</span>(<span class="hljs-params">vector,theta</span>):</span>
    <span class="hljs-string">"""Rotates 3-D vector around z-axis"""</span>
    R = np.array([[np.cos(theta), -np.sin(theta),<span class="hljs-number">0</span>],[np.sin(theta), np.cos(theta),<span class="hljs-number">0</span>],[<span class="hljs-number">0</span>,<span class="hljs-number">0</span>,<span class="hljs-number">1</span>]])
    <span class="hljs-keyword">return</span> np.dot(R,vector)
</code></pre><p><strong>Rotate Blue Vector 45 degrees (pi/2)</strong></p>
<pre><code><span class="hljs-attribute">L_predef</span> = np.array([<span class="hljs-number">11</span>.<span class="hljs-number">231303753070549</span>, <span class="hljs-number">9</span>.<span class="hljs-number">27144871768164</span>, <span class="hljs-number">18</span>.<span class="hljs-number">085790226916288</span>]) #blue vector
<span class="hljs-attribute">new_vect</span> = z_rotation(L_predef, np.pi/<span class="hljs-number">2</span>.<span class="hljs-number">0</span>)

<span class="hljs-attribute">fig</span> = plt.figure()
<span class="hljs-attribute">ax</span> = fig.gca(projection='<span class="hljs-number">3</span>d')
<span class="hljs-attribute">ax</span>.plot(np.linspace(<span class="hljs-number">0</span>,L_predef[<span class="hljs-number">0</span>]),np.linspace(<span class="hljs-number">0</span>,L_predef[<span class="hljs-number">1</span>]),np.linspace(<span class="hljs-number">0</span>,L_predef[<span class="hljs-number">2</span>]))
<span class="hljs-attribute">ax</span>.plot(np.linspace(<span class="hljs-number">0</span>,new_vect[<span class="hljs-number">0</span>]),np.linspace(<span class="hljs-number">0</span>,new_vect[<span class="hljs-number">1</span>]),np.linspace(<span class="hljs-number">0</span>,new_vect[<span class="hljs-number">2</span>]))

<span class="hljs-attribute">plt</span>.show()
</code></pre><p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1659839494124/75oYHwSaw.png" alt="image.png" /></p>
]]></content:encoded></item></channel></rss>