Clojure microservices for JavaScript developers part 2

This series was co-written by Tyler Sullberg and Musa Barighzaai.

In the previous post, we explored high-level differences between thinking in Clojure compared to thinking in JavaScript. We are now ready to start building our first Clojure microservice.

The microservice we are going to build will be very simple. It will be an HTTP server that uses a Redis data store to count how many times a given IP address has pinged the /counter endpoint.

We will walk you through each step of how to build this service in this tutorial. If you would rather, you can find the completed project here.

Please note that when you are developing in Clojure, you will mostly be working in the REPL, but for the sake of clarity, we won’t use the REPL in this tutorial. We highly recommend that you check out these instructions for using the REPL and start making the REPL central to your Clojure development workflow.

Before we can start building out the project, we should introduce ourselves to the tool that will manage our project structure, Leiningen.

Using Leiningen to automate Clojure project management

Leiningen is a tool for automating aspects of Clojure project management. Leiningen is the most popular tool for setting up Clojure projects. We are particularly fond of it at CircleCI because it was built by our very own Phil Hagelberg.

The closest analogs to Leiningen in the JavaScript world are Yarn and npm. Like Yarn and npm, Leiningen is how you will manage dependencies and publish packages. There are important differences in how Leiningen works, though.

• Leiningen is opinionated about your project structure.
• Leiningen will install dependencies (Maven under the hood) to a common location on your machine that can be used by multiple projects. Yarn and npm install dependencies into the node_modules folder of each project.
• Leiningen runs your tests and configures your REPL.
• Leiningen creates your Uberjar file. Your Uberjar file is a stand-alone executable JAR file that contains your project and its dependencies.
• And more.

Setting up your first Clojure project

First, download Leiningen.

Next, we will create our Leiningen application clojure-for-js-devs

Lang:shell
lein new app clojure-for-js-devs

Now open the clojure-for-devs directory in an editor and you will see that Leiningen has built out a project for you.

Most of these files are self-explanatory - the src directory and test is where we will write our tests. Similar to ES or CommonJS modules, each .clj file we create in these directories will be its own namespace. The project.clj is the most important file of this project.

(defproject clojure-for-js-devs "0.1.0-SNAPSHOT"
 :description "Simple app to demonstrate how to construct Clojure microservices."
 :url "https://github.com/tsully/clojure-for-js-devs"
 :dependencies [[org.clojure/clojure "1.10.1"]]
 :main ^:skip-aot clojure-for-js-devs.core
 :target-path "target/%s"
 :profiles {:uberjar {:aot :all
                      :jvm-opts ["-Dclojure.compiler.direct-linking=true"]}})

project.clj is very similar to a package.json file:

• dependencies is where we will bring in Clojure/Java dependencies. This is the same as dependencies in a package.json.
• main is the entry point for your application. When you start a Lein application with lein run, it will run the main function in whichever namespace you specify here. There is already a main function in the clojure-for-js-devs.core namespace. This is functionally the same as the main key in a package.json.
• target-path is the directory where compiled files will be placed when you run lein uberjar to build your production application. If you have used webpack before, this is similar to the output key instructing webpack where it should output your bundles.
• profiles allows you to customize your project.clj setup for development, production, or the REPL. Profiles allow you to customize any aspect of your project.clj. You can bring in specific dependencies, include additional directories, and change compilation options, among other things. Our project.clj has only one profile, uberjar, which is used for compiling our production application. You can add other profiles like dev and repl. There is not a great comparison for “profiles” in Node.js. To recreate similar functionality in Node.js, you often need to use a combination of environment variables likeNODE_ENV), along with scripts, devDependencies, and config files to get the job done.

You can also create custom scripts in the same way that you can create npm scripts, but we will cover that in the next post.

Note that the configuration of the project.clj is wrapped in a defproject. Defproject isn’t a new Clojure construct you need to learn, it’s just a macro from Leiningen. This is why the body of defmacro seems to defy the normal Clojure syntax that we’ve grown to know and love.

Creating the components of the application

Now that we have the shell of our application, we’re ready to start creating the major components, our HTTP server and Redis client.

In JavaScript, to create our Redis client, we’d do something like:

const redis = require("redis");
const client = redis.createClient();

client.on("error", function (error) {
  console.error(error);
});

client.set("key", "value", redis.print);
client.get("key", redis.print);

And to create our HTTP server, we might do something like:

const express = require("express");
const app = express();

app.get("/", function (req, res) {
  res.send("Hello World");
});

app.listen(3000);

In other words, to create these components, we end up creating a giant object that has all of the methods and properties we need to conveniently manage the lifecycle of that component.

Of course, since Clojure is a functional language, our approach is going to look different:

• Instead of burying our functions as methods in objects, the functional approach is to define functions that can be freely composed and reused.
• Instead of defining how an object should behave via interfaces and inheritance, Clojure uses polymorphism.
• Instead of creating state, Clojure will (generally) use pure functions and immutable data structures.

Before we can show the functional approach for building the HTTP server component and Redis component, we need to give a bit of background about two Clojure constructs that may be unfamiliar to you if you’re a new Clojure developer: defprotocol and defrecord.

Learning how these concepts work will be crucial for understanding how your microservice is structured.

Understanding defprotocol and defrecord

defprotocol defines a set of methods along with their function signatures. Protocols do not have an implementation, but any data type can implement that protocol and define the implementation for each of the protocol’s methods. extend-protocol is one way of defining the implementation of a protocol for a data type.

(defprotocol Bird
 (eat-bread [this bread-type]))

(extend-protocol Bird
 java.lang.String
 ; the implementation of the eat-bread function if called with a string
 (eat-bread [name bread-type]
   (println (str name " is eating my " bread-type " loaf")))
 ; the implementation of the eat-bread function if called with a long
 java.lang.Long
 (eat-bread [number-of-birds bread-type]
   (println (str number-of-birds  " birds are eating my " bread-type " loaf"))))

(eat-bread "Andrew Bird" "sourdough")
; Andrew Bird is eating my sourdough loaf
(eat-bread 3 "sourdough")
; 3 birds are eating my sourdough loaf

In this example, we create a protocol called Bird which defines the function signature for eat-bread. Notice how the first argument of eat-bread is this. The this in the defprotocol is superficially similar to “this” in JavaScript. Iit represents the data that is implementing this function. ‘This` isn’t a protected keyword in Clojure. We could name that first parameter anything.

When we use extend-protocol on Bird, we’re creating the actual function implementations of the Bird protocol. These function implementations will be different based on the data type of the first argument (i.e. this). This is polymorphism. Different function bodies are executed based on the data type of the first argument. If it’s a string, we execute one function body, and if it’s a long, we execute another.

So, what if we want to extend the Bird protocol with our own data type rather than a primitive type? We can create our own data type with defrecord. defrecord creates a Java class that can have properties and implementations of protocols. We can use the -> constructor to make an instance of this class. Let’s make a defrecord for a Duck:

(defrecord Duck [name])

(def reggie-the-duck (->Duck "Reginald"))
(println reggie-the-duck)
; #clojure_for_js_devs.core.Duck{:name Reginald}

To give our Duck the ability to eat-bread and make-noise, we can use the Bird protocol. We can use extend-protocol to define how the function signatures defined in the Bird protocol should be implemented for the Duck data type. A better way to implement the protocol is to define the protocol’s implementation directly on the defrecord.

(defprotocol Bird
 (eat-bread [this bread-type])
 (make-noise [this repeat-num]))

(defrecord Duck [name]
 Bird
 (eat-bread [_this bread-type]
   (println (str name " is eating my " bread-type " loaf")))
 (make-noise [_this repeat-num]
   (println (repeat repeat-num "Quack! "))))

(def reggie-the-duck (->Duck "Reginald"))

(eat-bread reggie-the-duck "sourdough")
; Reginald is eating my sourdough loaf
(make-noise reggie-the-duck 3)
; Quack!  Quack!  Quack!

defrecord and defprotocol will be central to how we structure our microservice, so learning these concepts now will give you much more confidence in your ability to build Clojure microservices.

Using stuartsierra/components

The stuartsierra/component library will be the heart of how our application manages the runtime state of our Redis client and HTTP server components. stuartsierra/component is a framework for managing the lifecycle of components. It makes sure components are stopped and started in the right order and explicitly declares the shared state between components. This framework allows us to make sure that our Redis client is started before our HTTP server, and that the HTTP server knows how to interact with Redis. Components will seem a lot like OOP objects, but remember that we’re still living the Clojure paradigm of pure functions and immutable data structures.

Setting up Redis

Now we can created our first component, the Redis client. Go to your project.clj file and add stuartsierra/component and carmine, a Clojure Redis client, to the dependencies.

 :dependencies [[org.clojure/clojure "1.10.1"]
                ; Add the following two dependencies
                [com.stuartsierra/component "0.4.0"]
                [com.taoensso/carmine "2.20.0"]]

Next, create a redis.clj file in your src/clojure_for_js_devs directory with the following code.

(ns clojure-for-js-devs.redis
 (:gen-class)
 (:require [com.stuartsierra.component :as component]
           [taoensso.carmine :as car]))

(defrecord Redis [uri connection]
 component/Lifecycle
 (start [this]
   ; If there's already a connection return the instance of the Redis class
   (if (:connection this)
     this
     ; Otherwise, associate the 'connection' property of this defrecord with
     ; a map representing the Redis connection
     (do
       (println "Starting Redis component")
       (println "Redis connection URI" this)
       (assoc this :connection {:pool {} :spec {:uri uri}}))))

 (stop [this]
   (if (:connection this)
     (do
       (println "Stopping Redis component")
       (assoc this :connection nil))
     this)))

(defn new-redis
 "Create instance of Redis class"
 [uri]
 (map->Redis {:uri uri}))

Let us break down what’s going on here. We’re creating a defrecord called Redis and a function to create a new instance of Redis that takes a URI as its only argument. In the Redis component, we implement a protocol called component/Lifecycle. This protocol comes from the stuartsierra/component library and has two function signatures: start and stop. The stuartsierra/component will call the start and stop function on our new Redis defrecord to manage this component’s lifecycle.

So far, this component just keeps track of a Redis connection, but doesn’t actually have any functions for interacting with it. Instead of defining these functions inside of the Redis class, like we might with an object-oriented language, we’re going to define separate functions that can take a Redis class as an argument. Add these these to the bottom of your redis.clj file.

(defn ping
 "Check that Redis connection is active"
 [redis]
 (car/wcar (:connection redis) (car/ping)))

(defn getKey
 "Retrieve count for a key in Redis DB."
 [redis key]
 (car/wcar (:connection redis) (car/get key)))

(defn incr
 "Increment count for a key in Redis DB."
 [redis key]
 (car/wcar (:connection redis) (car/incr key)))

We need to also create a docker-compose-services.yml file in the root directory so that we can run a Redis server during development

version: "2"
services:
  redis:
    image: redis:4.0.2-alpine
    ports:
      - "127.0.0.1:6379:6379"

Creating the component system map

Next, we need to create a “component map” that will describe how our components, including Redis, will interact with each other. This component map will initially include only Redis, but will add in the HTTP server component afterwards. Move over to the src/clojure_for_js_devs/core.clj file and copy/paste the following code.

(ns clojure-for-js-devs.core
 (:gen-class
  :main true)
 (:require
  [com.stuartsierra.component :as component]
  [clojure-for-js-devs.http :as http]
  [clojure-for-js-devs.redis :as redis]))

; Create a variable called *system* that can only be defined once
; ^:dynamic means that this variable can be rebound to a new value
(defonce ^:dynamic *system* nil)

(defn main-system
 "Creates map of component dependencies with implementation of Lifecycle protocol"
 []
 (component/system-map
  :redis (redis/new-redis "redis://redis:6379")))

(defn start
 "Start components of system in dependency order. Runs SystemMap implementation of Lifecycle protocol's 'start' function"
 [system]
 (try
   (component/start system)
   (catch Exception _ex
     (println "Failed to start the system"))))

(defn stop
 "Stop components of system in dependency order. Runs SystemMap implementation of Lifecycle protocol's 'stop' function"
 [system]
 (component/stop system)
 ; dynamically rebind *system* var back to nil
 (alter-var-root #'*system* (constantly nil)))

(defn -main
 "Entry point to the application."
 []
 (let [system (start (main-system))]
   ; dynamically rebind *system* to the newly created SystemMap instance
   (alter-var-root #'*system* (constantly system))
   ; Create hook that stops the component system in a controlled manner before the JVM completely shuts down
   (.addShutdownHook (Runtime/getRuntime) (Thread. (partial stop system)))))

If you’re interested in the details of what’s going on here, the stuartsierra/component source code is a short and useful read. Here is a quick overview:

• When the application starts, the main function will be automatically called (see: the :main key of the project.clj)
• The main function will create a SystemMap (remember, an immutable data structure, not an object) that represents all of the dependency relationships between the components of the application.
• Under the surface, the SystemMap is implemented as a defrecord that implements the Lifecycle protocol’s stop and start methods. It does so in the same way that our Redis defrecord implemented the Lifecycle protocol.

The last line deserves further explanation:

(.addShutdownHook (Runtime/getRuntime) (Thread. (partial stop system)))

We’re creating a shutdown hook here to detect when the application is shutting down, so that we can cleanly stop our component system first. The . in addShutdownHook signifies that we’re using a Java method of the Runtime Java object. The . at the of Thread. signifies that we’re creating an instance of Java Thread class.

Setting up the HTTP server

Start off by adding some new dependencies to the project.clj.

  :dependencies [[org.clojure/clojure "1.10.1"]
                 [com.stuartsierra/component "0.4.0"]
                 [com.taoensso/carmine "3.1.0"]
                 [compojure "1.6.1"]
                 [ring/ring-core "1.8.0"]
                 [ring/ring-defaults "0.3.2"]
                 [ring/ring-jetty-adapter "1.8.0"]
                 [ring/ring-json "0.5.0"]]

Now, create a new http.clj file in the src/clojure_for_js_devs directory. We’ll start by making the defrecord of the server.

(ns clojure-for-js-devs.http
 (:require
  [com.stuartsierra.component :as component]
  [compojure.core :refer [GET routes]]
  [compojure.route :as route]
  [ring.adapter.jetty :as jetty]
  [ring.middleware.defaults :as ring-defaults]
  [ring.middleware.json :as ring-json]
  [clojure-for-js-devs.handlers :as http-handlers]))

; this is just a function stub - we'll build this out soon
(defn start-server
 "Start the HTTP server with our routes and middlewares."
 ([host port redis-component]))

(defrecord WebServer [host port]
 component/Lifecycle
 (start [this]
   (if (:server this)
     this
     (let [server (start-server host port (:redis this))]
       (assoc this :server server))))
 (stop [this]
   (when-let [server (:server this)]
     (.stop server))
   (dissoc this :server)))

(defn new-server [host port]
 (->WebServer host port))

The WebServer defrecord is built roughly the same way as the Redis deferecord we created earlier. WebServer implements the Lifecycle protocol and we’ve created a new-server function that makes an instance of the WebServer class. We haven’t yet built out the start-server function, but we will do that shortly.

There’s one important difference, however, between our WebServer and Redis defrecords. Notice how when start-server is called, we pass in (:redis this) as its third parameter. What’s going on here? Where did :redis come from? If this refers to an instance of the WebServer class, then when did a :redis key end up in that data structure?

The :redis key is going to be added into this component by the stuartsierra/component library. Go back to core.clj file and modify the system map so that our WebServer can know about the Redis component.

Modify the imports and main-system function of your core.clj file with the following code:

(defn main-system
 "Creates map of component dependencies with implementation of Lifecycle protocol"
 []
 (component/system-map
  :redis (redis/new-redis "redis://redis:6379")
  :http-server (component/using
                (http/new-server "0.0.0.0" 8080)
                [:redis])))

We’ve added our HTTP server as another component in our system map, and we’ve also made the Redis component a piece of state that our HTTP server will have access to under the :redis key.

Head back over to http.clj so we can create our routes and middleware. To do this, we’re going to use a few libraries:

• Jetty is Java HTTP server. We’re using Jetty via a Ring library that adapts Jetty for Clojure.
• Ring is a collection of libraries that creates our Jetty server and defines the middleware for processing requests and sending responses.
• Compojure is a routing library for Ring.

Now we’re ready to create our routes. At the top of http.clj, but below where we import in external namespaces, add the following snippet.

(defn app-routes
 "Creates Ring handler that defines routes and route handlers"
 [redis-component]
 (routes
  (GET "/hello-world" [] (http-handlers/hello-world-handler))
  (GET "/ping" [] (http-handlers/ping-handler redis-component))
  (GET "/counter" req
    (http-handlers/counter-handler req redis-component))
  (route/not-found "Not found")))

The app-routes function will take in a redis-component as an argument which it will then pass to its route handlers. Go ahead and create the route handlers. Make a new file handlers.clj in your src/clojure_for_js_devs directory, and add the following code.

(ns clojure-for-js-devs.handlers
 (:require [clojure-for-js-devs.redis :as redis]))

(defn hello-world-handler
 "To check that HTTP server is working."
 []
 "howdy!")

(defn ping-handler
 "To check that HTTP server can interface with Redis."
 [redis-component]
 (println "Handling ping request")
 (redis/ping redis-component))

(defn counter-handler
 "Increment count of times that IP address has hit endpoint and return count."
 [req redis-component]
 (let [ip (:remote-addr req)
       counter (redis/getKey redis-component ip)]
   (redis/incr redis-component ip)
   (str "Counter: " counter)))

Now that we have our HTTP handlers, go back to the http.clj file. So far, we’ve defined our routes, but we haven’t yet actually connected these routes to our HTTP server, defined how incoming requests will be processed, or even started our Jetty server. Replace the function stub we made for start-server with the following function.

(defn start-server
 "Start the HTTP server with our routes and middlewares."
 ([host port redis-component]
  (-> (app-routes redis-component)
      ;; Turn map response bodies into JSON with the correct headers.
      (ring-json/wrap-json-response)
      ;; Turn JSON into map
      (ring-json/wrap-json-body {:keywords? true})
      ;; Parse query strings and set default response headers.
      (ring-defaults/wrap-defaults ring-defaults/api-defaults)
      (jetty/run-jetty {:host host
                        :port port
                        :join? false}))))

It is helpful to understand what the -> is doing here. -> is a called threading macro. Remember how in the first post we said that a macro lets you extend the Clojure language to things that otherwise wouldn’t syntactically be possible? Well, that’s what’s happening here. This threading macro works by executing the first function after the -> and then passing the result to the next function in the list. You could accomplish the same thing with a bunch of nested function calls, but this looks prettier, right?

start-server is defining how an incoming request will be processed. It begins by defining the routes, then defines how request is processed, then finally starts the jetty server.

Our microservice is now functional and ready to go! Fire it up:

docker-compose -f docker-compose-services.yml up
lein run

Go to http://localhost:8080/counter in your browser to see if the routes work.

As you refresh /counter, it should increment.

Wrapping up

Congratulations! You’ve just created your first Clojure microservice. You’ve learned the basics of how we structure our application with Leiningen and the stuartsierra/components library. You’ve also learned the key Clojure concepts defrecords and defprotcols. You now know about macros that are central to how these libraries work.

We’re not done yet, though. In the next post, we’re going to build out testing and continuous integration (CI) for our new microservice.