The race for faster CI/CD operations has been the central focus for most software companies that provide devops solutions. Google Cloud Platform currently sits at the top of the list when it comes to seamless continuous integration and delivery.
I am going to talk about how we can deploy a simple Pub/Sub Python application in Google Cloud Kubernetes Engine. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Now Google has built a solution thats sits on top Kubernetes and it has revolutionized the use of containers for enterprise applications that require a clustered distributed architecture, to ensure that services are highly scalable and robust during future upgrades and rollouts and even during application failures using various self-healing mechanisms.
Google Cloud Pub/Sub is a message bus service for delivering trillions of queued messages between various applications. Pub/Sub offers us the ability to provision such a service in a clustered environment that can auto-scale the number of nodes running either up or down based on the number of requests flowing through
To get started, follow this steps to setup the Kubernetes Engine API:
- Visit the Kubernetes Engine page.
- Create or pick a project.
- Make sure you enable billing for the project.
- Enable the Pub/Sub API.
- Install the Google Cloud SDK, which ships with the gcloud CLI tool.
- Download and install Docker Community Edition to build container images.
- Ensure you have Git running in your machine.
- Use gcloud from your terminal to install the Kubernetes CLI tool, which will enable you to interact with Kubernetes. Run this command:
gcloud components install kubectl
Create a container cluster called python-pubsub-cluster that will contain one or more nodes running the Python application. Run this command:
gcloud container cluster create python-pubsub-cluster
Create a Pub/Sub topic and subscription, which are used to publish and pull messages from the Pub/Sub service. Run these commands:
gcloud pubsub topics create my-topic gcloud pubsub subscriptions create my-topic-read --topic=my-topic
Clone this Python application on Github on your local machine.
Applications to be deployed in the Kubernetes Engine should contain yaml files in a manifest folder. These yaml files are used to configure the application’s deployment and other services such as an Ingress, Load Balancer or a Node Port service that provides an external IP Address for the application, as well as routing requests to the various containers/pods running.
Perform the following:
cd cloud-pubsub/deployment/ kubectl apply -f pubsub.yaml
To view the pods deployed and running:
kubectl get pods -l app=pubsub
You can also view the logs generated by the running pods:
kubectl logs -l app=pubsub
Kubernetes Engine runs Docker containerized applications, which it refers to as pods. To build a Docker image you need to create a Dockerfile, which contains a series of build instructions to install all the dependencies of the application and run a specific service once its deployed.
Create a service account and download the generated private key as a JSON. Service accounts enable us to provision role-based accounts with security keys to perform authentication, while the application is interacting with the Pub/Sub API using the provided client library.
You will need to import the generated credentials as a Secret into kubernetes using this command:
kubectl create secret generic pub-sub-key --from-file=key.json=<path-to-key-file>.json
A secret will be created in the Kubernetes Engine and will used in the deployed applications.
To use the secret in the application, modify the pubsub.yaml file with the following:
volumeMounts:- name: google-cloud-key mountPath: /var/secrets/google env: - name: GOOGLE_APPLICATIONS_CREDENTIALS value: /var/secrets/google/key.json
Deploy the modified manifest by running this command:
kubectl apply -f pubsub.yaml
Publish some Pub/Sub messages and try to read them as a test, use this commands:
gcloud pubsub topics publish my-topic --message="My first Pub/Sub message!"
To view if the message was published and pulled by the application, view the logs:
kubectl logs -l app=pubsub
To delete the topic, subscription, and cluster create run the following commands:
gcloud pubsub subscriptions delete my-topic-read gcloud pubsub topics delete my-topic gcloud container clusters delete python-pubsub-cluster
Tip 1: To understand Kubernetes in a less technical but more comical way, check out this funny comic piece created as a marketing pitch by Google 🙂
Tip 2: To drain out or force acknowledge all published messages, while testing your application, as you might not want to see certain unacknowledged messages again, use the following command:
gcloud alpha pubsub subscriptions seek projects/<project-name>/subscriptions/my-topic-read --time=2018-08-27T12:40:00
Of course change the –time to the current time to drain all messages up till date 🙂