When it comes to choosing passion projects and side studies, CircleCI engineers have considerable freedom. One of our support engineers, Zachary Scott, has been spending some of his free time tinkering with Elixir, a dynamic, functional language. It has enough in common with Clojure — our language of choice here at CircleCI — that we thought folks would enjoy a deep dive into this (relatively) new language. Enjoy!

It’s probably no surprise that we’re excited about Elixir, a functional language built on Erlang VM that shares many of Clojure’s philosophies. Before I talk about Elixir’s design and why it’s so amazing, I’d like to share how I stumbled on this beautiful language.

April Fools: My Introduction to Elixir

If you’re familiar with Erlang, Ruby, or functional programming in general, you may have already heard about Elixir. Fortunately for me, I live under a rock in Japan, so there wasn’t a chance to try Elixir until I heard that José Valim — Elixir’s creator and benevolent dictator for life (BDFL) — was visiting.

I met José four years ago when I attended Ruby Kaigi in Japan. I had been contributing to Ruby, and he was giving the first conference keynote. There, he discussed concurrency in Ruby and shared his hard work fixing thread-safety bugs as part of his work as Rails core team member.

About a year a later, Elixir hit 1.0, and the first ElixirConf was held. Since then, Elixir has grown more popular, with the release of several books and more conferences around the globe. On April 1, 2017, I attended the first ElixirConf held in Japan.

José opened the conference with an elegant tale of his struggle to find the exact ingredients to make his language, starting in 2011.

Being one of 300 eager attendees watch José recount his journey was satisfying all the way from the back row. José also discussed his plans for the language, including UTF-8 Atom support, a new GenHTTP concept, and an adding property testing a la QuickCheck to core.

Daniel Perez enlightened newbies such as myself on the basics of Elixir and its style of Functional Programming. This talk was exactly what I needed to understand the core concepts of Elixir and how it’s designed.

Keiji Rikitake spoke about the recent Erlang and Elixir Factory conference in San Francisco and encouraged the growing Japanese Elixir community to go forth and share their knowledge with the world. In the final keynote, voluntas dropped massive amounts of knowledge from his extensive background hacking on Erlang and OTP.

There were so many amazing sessions that, by the end of the day, my head was spinning and exploding with new ideas about programming.

It was also interesting to see so many familiar faces from the Ruby community at the conference. José has attracted many programmers from across the globe with his new language, including Koichi Sasada, the author of Ruby’s modern Virtual Machine used today, who helped translate Dave Thomas’ Programming Elixir into Japanese.

Why Elixir?

So, Elixir has a lot going for it; the community is full of really talented people and is growing and that is great to see. But what about the language itself? Why are all of these people, including me, interested in it?

If we look at the core goals of the language, we’ll see that it has a lot in common with Clojure. And that’s great because, until recently, Clojure was one of the few modern languages that was perfect for building highly concurrent and scalable applications. Now we have a second option that isn’t tied to the JVM. Instead, Elixir is built on top of the Erlang VM, and this has many benefits which we’ll cover shortly.

Language Goals

For starters, Elixir focuses on programming productivity. This generally means that getting started and developing software in the language should be easy for all experience levels.

Getting Started is Easy

We can easily get started with Elixir in a few minutes after setting up the host Erlang environment. Below are some brief instructions for setting up Elixir on Ubuntu:

wget https://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc
sudo apt-key add erlang_solutions.asc
sudo apt-add-repository "https://packages.erlang-solutions.com/ubuntu $(lsb_release -sc)
 contrib"
sudo apt-get update

Once we’ve authorized the Erlang Solutions repository, we can install Elixir:

sudo apt-get install elixir

We can verify that Elixir was installed by using the --version flag:

$ elixir --version
 
Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [kernel-poll:false]
 
Elixir 1.4.5

We’ve done it!

Now we should have a working Elixir environment!

Documentation is First-Class

Documentation is a first-class citizen of Elixir via Doctests, so it’s plentiful throughout the core language and ecosystem.

Doctests can be added to your Elixir code and executed along with your test suite in order to ensure the documentation is accurate. One of the biggest problems with writing docs without this ability is that, once it’s written, you’ll have two implementations of the code you’re documenting: the one you wrote, and the comments which describe its behavior. This can lead to increased complexity and maintenance burden, confusing future generations as these comments go out of date.

You can read this documentation at anytime inside of IEx, Elixir’s Read-Eval-Print-Loop (REPL). For example:

Interactive Elixir (1.4.2) - press Ctrl+C to exit (type h() ENTER for help)
 
iex(1)> h Enum
                                  	Enum                                 	 
 
Provides a set of algorithms that enumerate over enumerables according to the
Enumerable protocol.
 
	iex> Enum.map([1, 2, 3], fn(x) -> x * 2 end)
	[2, 4, 6]
 
...

Productive Tooling Abounds

Elixir provides more than just documentation and a REPL; it also has a test unit framework called ExUnit. There’s also the Elixir build tool Mix, which handles everything from the generation of your project to a starting layout. It also deals with compiling your project down to bytecode to run on the Erlang VM (BEAM) and provides tasks for running test suites and configuring your project’s environment.

There’s also a myriad of packages available through Hex, the official package manager for Elixir (and Erlang!).

This sands the edges of learning a new language, including installation, dependencies, and project organization/structure.

Host Compatibility

Much like Clojure, Elixir is designed to maintain compatibility with its host. This means it shares the foundations of its host platform, including: native data types, garbage collection, and a concurrency system native to the host VM — a staple of the Erlang ecosystem.

With the help of hex, we can take advantage of the huge ecosystem already supported in the Erlang community. This benefit is similar to what Clojure gets by being hosted on the JVM, including all of Java’s great libraries and tools. The ability to reach into the Erlang runtime interoperability is a huge feature.

Macros For The Win

Lastly, Elixir has a clear focus on extensibility. This includes the ability to create macros as a form of meta-programming and is the foundation on which the language is actually defined.

If you’re not familiar with macros, they’re used to extend the language at compile time, in order to define program behavior as if it was part of the language itself.

It’s a powerful concept, but macros allow you to design your system from the “bottom-up”, which translates to building the language up toward your program as opposed to building down, or on-top-of the language. This means we can extend our application in terms of the problem domain or business model.

In Elixir, José wanted macros similar to Lisp but with an obvious syntax. As an example, let’s take a look at Elixir’s quote macro, which we can use to get the internal representation of a function call.

quote do: sum(1,2,3)
{:sum, [], [1,2,3]}

We can see that the first element is the function name (sum), followed by metadata ([]), and a list of arguments ([1,2,3]).

Since Elixir also supports optional parentheses, let’s see what happens if we call sum() without them.

quote do: sum 1, 2, 3
{:sum, [], [1,2,3]}

That’s right; the parens are just syntactic sugar and still generate an identical representation of our code.

How about operators?

quote do: 1 + 2
{:+, [], [1, 2]}

When we want to inject certain data into a representation, we can use unquoting.

quote do
  def hello() do
    unquote(value)
  end
end

If you’re familiar with macros in Clojure or Lisp, you’ll notice that Elixir supports an explicit syntax for quoting. There is no shortcut syntax to quote and unquote, such as ‘() or back-tick.

But it doesn’t stop there.

In order to use a macro from another module in Elixir, you’ll need to explicitly call require or use. This is intended to prevent abusing macros to modify global state and is a design choice of Elixir to promote responsible meta-programming.

Concurrency

Finally, I’m really excited about Elixir’s focus on concurrency as a first-class feature. This was clearly one of the reasons why José decided to use Erlang as the host environment, after working so hard to improve thread-safety in Ruby and Rails.

First, all of Elixir’s native data types — Strings, Collections, even Maps — are all immutable. This means that, once any of these are created, they can’t be changed. Any actions you take (like concatenating two strings) will never modify the original, instead returning a new value. By relying on this fact in Elixir, we can avoid an entire category of bugs that only occur when writing concurrent or parallel programs. No more race conditions or deadlocks!

Second, Elixir can take advantage of Erlang’s native concurrency model. In order to achieve true concurrency, you need a model that can handle several threads of execution simultaneously. This allows us to write programs that fully utilize all provided cores and threads, whereas traditional languages like C are designed with a single-threaded model in mind.

What makes Elixir so special is that it can wrap lightweight processes from the Erlang VM and distribute execution across up to thousands of processes using a more natural syntax. Elixir also provides syntax for message passing to communicate among multiple processes, and abstractions for maintaining state shared among those processes — it even has an abstraction on top of background tasks that can be lazily executed.

Now let’s talk a bit more about some of the unique conventions Elixir provides in the language itself.

Elixir in Practice

Elixir is a functional programming language, but what does that mean exactly?

It means functions are first-class citizens; they’re treated like any other value, meaning they can be assigned to variables and passed around in the same way.

Higher-Order Programming

Elixir supports higher-order functions, which is just a fancy way of describing functions that accept other functions as parameters. Higher-order functions are useful for when we want to abstract common patterns. For example, the functions sum and count could be implemented like so:

def sum([head | tail]), do: head + sum(tail)
def count([head | tail]), do: 1 + count(tail)

We skipped the function definition (which takes an empty list) just so you get the idea. In any case, both functions follow these rules:

  • They recurse on a list.
  • They accumulate the result by applying a function.

These rules also apply to the function known as reduce or fold. We can implement our own reduce like so:

def reduce([head | tail], start_value, func) do
  func.(head, reduce(tail, start_value, func)
end

We could use this function to implement the sum and count functions:

def sum(list), do: reduce(list, 0, fn (element, increment) -> element + increment end)
def count(list), do: reduce(list, 0, fn (_element, increment) -> 1 + increment end)

You could argue that this implementation is more complex than our first implementation, and you’d be right. However, its purpose is to illustrate that we can define a function, and pass it while executing the body in the context of calling another function. This is thanks to higher-order programming.

Some other common examples of higher-order functions are map, which applies a function on all elements of a list, and filter, which filters a list using a predicate or boolean function.

Recursion

We saw a recursive function (reduce) just a moment ago, but let’s talk about why recursion is a key ingredient to Elixir’s success. All you need to know about recursive functions is that they call themselves, directly or indirectly.

Remember the sum function?

def sum([head | tail]), do: head + sum(tail)

This is a great example of a recursive function that takes the head from a list and adds that number to the rest of the list, so that each iteration is increasing the value until we reach the end of the list. This is a really common pattern in functional languages, especially if you have singly-linked lists, like Elixir.

No article about functional programming would be complete without the factorial function, so let’s do that one, too:

def fact(num) when num <= 1, do: 1
def fact(num), do: num * fact(num - 1)

The first clause just returns the number 1, but the second calls itself and passes a decremented number for the next iteration.

The second thing we should discuss is that these sorts of recursive functions can be highly inefficient, due to the number of triggered calls that end up on the stack. To correct this problem, we have to make sure the recursive call is the last step in the evaluation. This is also called a “tail call”. Functional languages are often optimized to eliminate duplicate calls on the stack until the tail call, compacting the call stack to avoid the infamous “stack overflow” exception. This solution is known as “tail-call optimization” (TCO).

If we want to make our sum function tail-recursive, we’ll have to use the second version, which used the higher-order function reduce along with it. The reason this matters is because recursion is used heavily to model loops; that is, we use iteration and enumeration when dealing with collections.

Pattern Matching

Pattern matching is another features that really excites me about Elixir. This is a powerful way to handle structured data by matching a given expression to another — even binding given values to target variables. Essentially, the pattern is a partial specification for the form of a given data structure.

I didn’t discover pattern matching until I started researching ML. That’s MetaLanguage, not MachineLearning…for you hipsters out there. You may have heard this referred to as destructuring, and it’s actually already in JavaScript and Clojure, much to my surprise.

In any case, whenever you see the = operator in Elixir, it’s not being used for assignment but actually pattern matching and is called the “match operator”. Let’s take a look at some examples:

{foo, bar} = {42, "some string"}
foo # 42
bar # "some string"

Here, we’re assigning the values to foo and bar by matching the data from a tuple.

%{key: value} = %{key: "some value", other_key: "another value"}
value # "some value"

In this case, we’re pulling the value out of a map (a key-value pair) by matching the key.

[head | tail] = [1, 2, 3]
head # 1
tail # [2, 3]

We can also pattern match on lists, as well as match the head/tail of a list.

def my_function({foo, bar}) do: foo + bar

This last example shows how we can use pattern matching in a function clause. This pattern is ubiquitous throughout the language and is a really nice way to express what we want our programs to do.

Take, for example, if and else. If we wanted to implement them using pattern matching, we could do so simply:

case x < 1 do
  true -> "if branch"
  false -> "else branch"
end

This clearly expresses intent: any value less than 1 is true and will execute the “if branch”, otherwise it will be false and execute the “else branch”. It’s also interesting to note that this will work on any data type as every one of Elixir’s native types is comparable. This is useful because it means sorting algorithms can accept any data type and still work.

Another interesting use case is for refactoring conditionals. Take the following contrived example, where we’re trying to determine whether someone is legally allowed to have a beer:

def can_have_beer?(age, country) do
  if country == "america" do
    if age >= 21 do
      "ok you can have a beer!"
    else
      "nope!"
    end
  end
 
  if country == "japan" do
    if age >= 20 do
      "どうぞ! ビール飲んでください!"
    else
      "だめ!"
    end
  end
end

How would we write this using pattern matching?

def can_have_beer?(age, "america") when age >= 21, do: "ok you can have a beer!"
def can_have_beer?(age, "america") when age < 21, do: "nope!"
def can_have_beer?(age, "japan") when age >= 20, do: "どうぞ! ビール飲んでください!"
def can_have_beer?(age, "japan") when age < 20, do: "だめ!"

This implementation is cleaner, more expressive, and easier to understand. Hopefully, this gives you an idea of how powerful pattern matching can be and why I’m so excited to use it in a language like Elixir. It’s an incredibly powerful concept that, once you have it, you’ll want to take with you everywhere.

Elixir in the Wild

No language is complete without a web-framework. It’s the Killer App, the MVP, which proves your language is ready for production. It takes all of the best practices and use cases of the community and packages it up into a productive powerhouse to encourage adoption and grow the ecosystem.

Ruby was nearly 15 years old before Rails was released, and very few people had even heard of it. Even today, many still wouldn’t be able to tell you the difference.

Phoenix

For Elixir, that framework is Phoenix. It’s bottled up everything good about Elixir and shipped it to the masses. Now at version 1.2.0, Phoenix is quickly becoming a staple for building fast, stable web applications.

CircleCI 2.0

I’ve had the pleasure of hacking on Phoenix and Elixir in my spare time, leading to a demo project that is running on CircleCI 2.0. It’s a simple chat application that I built while following the Up and Running guides as well as the Channels guide. All the code is available on GitHub at CircleCI-Public/circleci-demo-elixir-phoenix.

If you’re just getting started, check out the project’s .circleci/config.yml, which should help you start building your Phoenix app on CircleCI 2.0:

version: 2.0
jobs:
  build:
	docker:
  	- image: circleci/elixir:1.4.2
  	- image: postgres:9.4.1
    	env:
      	- POSTGRES_USER=ubuntu
	working_directory: ~/circleci-demo-elixir-phoenix
	steps:
  	- type: checkout
  	- run: mix local.rebar --force
  	- run: mix local.hex --force
  	- run: mix deps.get
  	- run: mix test
 

Notice how little configuration I actually need to build my app. This is due in part to CircleCI Images and Docker. We maintain these as a subset of the official Elixir image, along with some additional tooling to make it easier to build on our platform.

A Practical Example

Building toy chat applications with Elixir is trivial, but what about an existing project that is actually being regularly deployed to production?

Enter firestorm, an open-source forum engine built on top of Elixir and Phoenix. It also uses a hip new front-end language called Elm, which you should definitely check out.

Here’s the configuration I used to get the application building:

version: 2.0
jobs:
  build:
	environment:
  	- AWS_ACCESS_KEY_ID: "XXXX"
  	- AWS_SECRET_ACCESS_KEY: "XXXX"
  	- AWS_S3_BUCKET: "XXXX"
  	- AWS_S3_REGION: "XXXX"
  	- MIX_ENV: "test"
	docker:
  	- image: circleci/elixir:1.4.2
  	- image: postgres:9.4.1
    	env:
      	- POSTGRES_USER=ubuntu
	working_directory: ~/firestorm
	steps:
  	- type: checkout
  	- run:
      	name: Install PhantomJS Dependencies
      	command: |
        	[ -f /usr/local/bin/phantomjs ] || sudo apt-get update
        	[ -f /usr/local/bin/phantomjs ] || sudo apt-get install -y fontconfig wget
  	- run:
      	name: Install PhantomJS
      	command: |
        	[ -f /usr/local/bin/phantomjs ] || wget -O /tmp/phantomjs.tar.bz2 https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2
        	[ -f /usr/local/bin/phantomjs ] || tar -xjf /tmp/phantomjs.tar.bz2 -C /tmp
        	[ -f /usr/local/bin/phantomjs ] || sudo mv /tmp/phantomjs-2.1.1-linux-x86_64/bin/phantomjs /usr/local/bin/phantomjs
  	- run: mix local.rebar --force
  	- run: mix local.hex --force
  	- run: mix deps.get --only test
  	- run: mix test

Of course, if I created a custom image that included PhantomJS, this would be much simpler, but you get the idea. Also, the AWS credentials can be anything, but I found they were required for the build to pass – so probably a patch to make those optional is in the making!

Conclusion

It really motivates me that Elixir has pulled all of these great ideas from the past and present to create a truly modern language. I hope this deep dive has given you some ideas about the types of things you can do with Elixir. Below are some resources you can use if you’re trying to get Elixir set up with CircleCI:

Language guide for CircleCI 2.0
CircleCI Discuss Forum
Me on Twitter