Overview
Why would you want to run a scheduled Bash script in the cloud?
Couldn’t you just run cron on your local machine?
Running a bash script in the cloud rather than on a local machine offers developers a multitude of advantages that can enhance both
Efficiency — developers can ensure greater reliability and availability, as cloud services often offer redundancy and failover mechanisms that protect against data loss and downtime
Scalability — cloud environments can provide enhanced computational power and resources that can handle more complex or resource-intensive scripts, reducing execution time and improving performance.
To run bash scripts on a schedule in Google Cloud, you will need to setup the following components (detailed in the next sections):
Docker— The script with all of the dependencies. We will be packaging the Bash script and all required dependencies with a Dockerfile.
Cloud Build and Artifact Registry— Build the created Dockerfile into a Docker Image and push to Artifact Registry.
Cloud Run Jobs — Compute to execute the script
Cloud Scheduler — A service to trigger the bash script to run on a specified schedule.
Docker
The bash script and any dependencies it uses must be packaged into a dockerfile and then a docker container needs to be created as an artifact to be used in Cloud Run.
Lets first create a simple “hello world” bash script and save it as hello_world.sh
#!/bin/bash
echo "Hello, world! The time is $(date)."
Next, lets create a Dockerfile to run the bash script when the container is created. Save the filename as Dockerfile in the same directory as the bash script.
# Use a minimal base image with bash
FROM ubuntu:latest
# Set the working directory inside the container
WORKDIR /app
# Copy the bash script into the container
COPY hello_world.sh .
# Make the script executable
RUN chmod +x hello_world.sh
# Set the default command to execute the script
CMD ["./hello_world.sh"]
Cloud Build and Artifact Registry
Create a Docker repository in Artifact Registry
gcloud artifacts repositories create <YOUR_REPO_NAME> --repository-format=docker \
--location=us-west2 --description="Docker repository"
Build the docker image and push to Artifact Registry with Cloud Build
gcloud builds submit --region=us-west2 --tag us-west2-docker.pkg.dev/<PROJECT_ID>/<YOUR_REPO_NAME>/hello-world-image
If the docker image was built and pushed correctly, it should be in the output of this command
gcloud artifacts docker images list us-west2-docker.pkg.dev/<PROJECT_ID>/<YOUR_REPO_NAME>/
Cloud Run Job
To actually run our bash script contained in the Docker Image we created in the previous section, we need to deploy the image as a Cloud Run Job.
Unlike a Cloud Run service, which listens for and serves requests, a Cloud Run job only runs its tasks and exits when finished. A job does not listen for or serve requests.
Run the following gcloud command to deploy a container and create a Cloud Run job.
gcloud run jobs create my-script-job --image=us-west2-docker.pkg.dev/<PROJECT_ID>/<YOUR_REPO_NAME>/hello-world-image:tag1 --region us-west2
If done correctly, you should be able to execute the job and see the results with the command
gcloud run jobs execute my-script-job --region=us-west2
gcloud run jobs executions describe <execution_id> --region=us-west2
Cloud Scheduler
For the final step, we need to trigger our script to run on a schedule. We will be using Google Cloud Scheduler, which allows you to schedule and run a cron job to trigger HTTP/S endpoints and Cloud Run jobs.
To create a Cloud Scheduler Job to trigger the Cloud Run job we created in the previous section, run the following gcloud command.
gcloud scheduler jobs create http my-scheduler-job-1 \
--location us-west2 \
--schedule="0 * * * *" \
--uri="https://us-west2-run.googleapis.com/apis/run.googleapis.com/v1/projects/PROJECT-ID/jobs/my-script-job:run" \
--http-method POST \
--location=us-west2
The example cron schedule for this example is set to hourly at minute 0. Also, because the oauth-service-account-email argument was not explicitly specified in the command, it will use the default compute account for authentication to call the Cloud Run jobs API.
Conclusion
In conclusion, running a bash script in the cloud rather than on a local machine offers developers a multitude of advantages that can significantly enhance both efficiency and productivity.
By leveraging Docker, Cloud Build, Cloud Run, and Cloud Scheduler, developers gain access to unparalleled scalability, enabling them to run scripts on-demand without worrying about local resource limitations.
This scalability is complemented by the inherent flexibility of Google Cloud Platform, allowing developers to seamlessly integrate their scripts with a variety of tools and services, thereby streamlining workflows and reducing operational overhead.
References
https://cloud.google.com/run/docs/create-jobs
https://cloud.google.com/scheduler/docs/schedule-run-cron-job
https://cloud.google.com/build/docs