The importance of maintaining version dependencies for your microservice architecture
Scaling a microservice architecture has a number of challenges. You’ll hear people talk about having “thousands of services” where they get to “have the right tools for the job” all the way down to their languages. This is all well and good, but thousands of things are not always great.
For instance, things I would rather not have thousands of:
- Log formats
- Spider-man reboots
- Database access patterns
- Metrics dashboards I need to maintain
- Stats publishing formats
- Recipes including the words “Jello” and “Salad” (no, this was not okay, 1950s. You gave us the birth of NASA, but this almost cancels it out)
- Measurement standards
- Health check endpoints
- RabbitMQ access patterns
And at the top of the list: Java Maven Dependencies
While microservice architectures have well-known benefits like separation of concerns, limited blast radius, clarity on code boundaries, they also come with the overhead of maintenance, and commonly used code being repeated. One of the most common things teams do to improve collaboration is to start sharing “core” libraries. Everyone needs to map the same data-types from PostgreSQL, rotate the same credentials from Vault, prevent logging secrets from fat fingers, etc.
For our team, this core library is called Service Utils*. It bakes statsd, logging, and Rollbar support into pretty much everything you could possibly use to have your service do…well, anything. Boilerplate gets buried beneath the surface so now all log messages, statsd metrics, etc. have things like your service’s name, code version sha, K8s Pod Name available.
This has been huge for our team. We have a single Datadog dashboard called “Service Utils: Service X” which takes your service’s name, its database name, and viola! You have a full orchestra of a dashboard.
*this is a library which uses https://github.com/tolitius/mount to great effect, and if it weren’t for some grumpier-than-I fellow devs, I would have named it Mount-ain, or Stirrup…Yes, I’m still bitter about this.
Another super common thing is to have tens of versions of your core libraries in production (especially when you practice this wild thing called continuous delivery–I know, I know, it’s like I work for a CI/CD provider)
CircleCI is no different, and as we start setting out to solidify our dependencies (everything from service ssl certs to network traffic) myself, An Nguyen and Andres Cuervo set out to see just how many versions of Service Utils were kicking around in our production system.
To do this, we cooked up https://github.com/AlexanderMann/clj-deps. It is a simple brute force Clojure app which runs simply in CircleCI as a containerized Workflow Job.
We’re now using it periodically at CircleCI to generate a dependency graph and get a better understanding of what’s actually running in our production system. With ~2,000 distinct versions of only ~1,000 Maven libraries we have some conflicting versions of things…
The distribution looks something like:
For our core Service Utils projects we’re at 4 distinct versions of it kicking around. The number of projects which reference those 4 versions? Only 6!
“But wait, you said this was a core library used by all of your services? Does CircleCI only have 6 microservices in production?!?!”
The impetus for using this library has been carrot, not stick. We have tens of services, not thousands. Most of (all of?) our newest services use Service Utils. Which ultimately means that stepping into a service to debug something is a grab bag right now.
Using Clj-Deps, we hope to start pruning that massive dependency list down, and start identifying which services can be brought over to seeing the light.
As always, happy hacking!