Note from the publisher: You have managed to find some of our old content and it may be outdated and/or incorrect. Try searching in our docs or on the blog for current information.
Continuously deploying Clojure apps to Google App Engine
At CircleCI we write lots of Clojure. The main backend app is > 100,000 lines of Clojure code, and the frontend is > 30,000 lines of ClojureScript.
I love Clojure as a language, so much so that I decided to do my side projects in Clojure, too.
I was quite surprised when I realized that deploying to AWS requires a set of elaborate hacks to just get continuous delivery (push to master on GH -> production) working. Although I use AWS at work every day, I just can’t justify building my own implementation of things like secrets and blue / green deployments for my side projects. There is Kubernetes, but it’s currently not viable nor cost-effective to operate a k8s cluster just for my side projects (too much overhead). When I’m writing an MVP and don’t have time for infrastructure, what I need as a developer is good middle ground between terrible hacks for self-made CD process and Kubernetes.
I started looking at alternatives that would give me CD right out of the box with as little configuration as possible. While Heroku is a good option, I was looking for something a bit more flexible, and ideally affordable.
Google App Engine, as a fully managed app runtime, currently meets that criteria for me. GAE handles blue / green deployments and allows for more customization than what would be possible with Heroku. Google currently offers $300 in credit for everyone who decides to create a new GAE account, plus they offer a free tier. All those things in combination work out to be a good solution for putting out MVPs.
Changes to the default template to enable App Engine deployments
I spent quite some time getting to deploy my first Clojure app to GAE, so I figured I would create a skeleton project that folks could then use to push out their first projects to App Engine. I started out with a fresh-from-template Compojure project, and added only a few things.
-
Dockerfile. We are going to use the App Engine flexible environment, mostly because newer always means better, and Dockerfile is essential for getting started with that. The Dockerfile itself is very simple. All it does is start from the small Clojure base image, add the project code to
/codeand execute the entrypoint script. Ideally we would build a standalone uberjar and only put that into the container, but again, the purpose of this setup is to facilitate experimentation. -
docker-entrypoint.sh. Having the entrypoint script allows us to easily customise the command we need to start the service we are building. The Compojure lein template provides a convenient way to start the headless server with the app via
lein ring server-headless. Other frameworks or app types might need a simplelein runinstead. -
docker-compose.yml. The docker-compose file is there purely for a better development experience.
docker-compose up -dwill get the sample app up and running for you. -
.circleci/config.yml. The configuration for CircleCI workflows allows us to separate the testing step from the deployment. We also use two different base images for testing and for deployment—now that deploying to App Engine only means pushing the code out and letting Google Cloud do the rest for you, we don’t need the Clojure image when deploying. We use the image that provides the Google Cloud SDK instead.
Let’s go through the required steps to deploy a Clojure project to Google App Engine.
Continuously deploying a Clojure project to Google App Engine
We are going to assume that that you already signed up for a Google Cloud account and received the free credits kindly offered by Google.
1. Set up a Google Cloud project and enable Google App Engine Flexible environment
We’ll navigate to console.cloud.google.com, sign in or sign up, and then create a new Google Cloud project.
2. Create a service account and enable GAE API access
To give CircleCI deploy build permissions to push out new releases of our app, we will need to create a service account inside our project. The service account will need the App Engine Admin and App Engine flexible environment Service Agent permissions. When creating the service account, save the resulting JSON to disk.
Adding necessary permissions to the service account
Once that is done, we will need to enable two API sets for our project that are disabled by default.
In our project dashboard on Google Cloud, we can navigate to the API Manager item in the sidebar, then click Library, search for these items and click Enable on them:
- Google App Engine Admin API
- Google App Engine Flexible Environment
Enabling Google App Engine Admin API
3. Create the CircleCI project and add service account credentials
We have already pushed the code to our VCS provider, so now we navigate to the CircleCI Projects page and from there add our GitHub or Bitbucket repo as a new CircleCI 2.0 project. CircleCI will then run the build job once, and we’ll see that our tests are passing. The deployment step will not be triggered.
Now we’ll add the JSON with the service account we downloaded to the encrypted environment variables in our CircleCI project. We’ll call the variable GCLOUD_KEY_JSON. The config that’s present in the repo will output the contents of that variable into a file which we will then pass to the gcloud utility for authentication.
Adding the service account credentials to the encrypted env vars on CircleCI
4. Edit the project name in the CircleCI config and push it
We will now need to edit the project name in the gcloud calls in the CircleCI config to match the name of our project on Google Cloud. Once we commit and push our change, CircleCI will run the full build-and-deploy workflow and will get our app deployed.
A successful deploy job on CircleCI
From now on, merging pull requests on GitHub or Bitbucket will trigger both build and deploy jobs and continuously deploy our Clojure app to Google App Engine.
Gotchas
A few things in Google App Engine might annoy you or make your Google Cloud bill much larger than you would expect.
First, autoscaling is enabled by default. This is why in the app.yaml we select manual scaling and specify the instance type.
Second, the JVM cannot correctly interpret the memory limits set by the Docker daemon, no matter if you run the application locally or in Docker-based environments like Google App Engine (or CircleCI 2.0, for that matter). Java 8 doesn’t support cgroup-based memory limits. When you set a memory limit on a Docker container that’s running a Java 8 application, you are basically telling the Docker daemon to kill the container if the memory usage surpasses the limit, but the JVM is unaware of that limit. Therefore it allocates as much memory as is free on the machine it is running on (rather than the container). This results in consistent OutOfMemory errors. The solution is to limit the JVM memory usage explicitly by setting $JAVA_OPTS environment variable to contain, for example, -Xmx300m if you want the JVM to only use 300M of memory. The instance size specified in our app.yaml is the smallest possible instance (and the most affordable one), and that one only has 128MB of memory available. In the template project we’ve set the maximum Java heap to 100MB.
Conclusion
There are a few steps required to set up continuous deployment from CircleCI to Google App Engine for Clojure projects, but the process itself is quite straightforward.
For my personal use case—quickly deploying side projects—the biggest advantage of this setup is the fact that App Engine handles blue / green deployments automatically, so for smaller projects there is no need to shutdown your app before the deployment and start it up again once the deployment is finished (as I used to have to do in my CD scripts when I was deploying to an AWS EC2 machine). No more manual deployments, constantly-open tmux sessions on EC2 machines (true story), or undeployed changes.
