How Avant Runs 2.5 Million Daily Tests on CircleCI

The Challenge Prior to CircleCI

Avant is an online lending platform that is the leading provider of credit alternatives to middle income consumers in the US and UK. More than $3.6 billion in loans has originated on the Avant platform. With a rapidly growing business and the security needs a financial company requires, Avant needed an enterprise-grade continuous integration solution.

“On average we run roughly 2.5 million tests per day,” says Tim Mertens, Avant Test Engineering Manager. “Our largest test suite runs in parallel across 30 containers per build. One of the biggest software testing challenges for us is minimizing build times and build queuing. If our tests were run in serial, the test suite for our main application would take several hours to run. In a fast-paced environment like Avant where we do multiple production deploys every day, waiting a long time for builds to run is simply not acceptable. Long-running builds have significant ongoing compute costs, resulting in decreased productivity and longer feature development timelines. As such, we do everything we can to reduce our end-to-end build times.”

While the Avant team was familiar with the benefits that good CI/CD practices deliver, their previous CI tool did not have the flexibility they needed, and as a result Avant’s builds were slow and their infrastructure was expensive. Beyond that, they did not feel they had the transparency they expected from a tool running their code.

“Our previous CI provider was very protective about how they executed tests under-the-hood, to the point where they would not provide direct access to the commands they used to actually execute the tests,” says Mertens. “This sometimes made debugging intermittent test failures extremely difficult.”

CI is an integral part of the Avant team’s development and deploy process, and having the right tool in place is necessary for performing multiple deploys a day with confidence.

The Solution

Avant looked for opportunities to increase performance and security, while also reducing infrastructure cost. “CircleCI Enterprise enables us to execute on our business objectives more quickly, with more flexibility, and better cost structure than other hosted and enterprise CI providers we evaluated,” says Mertens.

“We chose CircleCI primarily for its ability to automatically parallelize tests across many workers. Another primary factor is its cost structure. The cost structure enables us to optimize infrastructure costs by provisioning spot instances on Amazon Web Services, instead of on-demand or reserved instances, and we can scale our compute up and down in response to demand. CircleCI’s robust API is also a plus, and has enabled us to implement time saving features–such as automatic bisect and time travel builds–which benefit the entire development team.”

Productivity

“We look for opportunities where we can achieve performance improvements that impact our test suite as a whole or where we can yield improvements across many tests at once,” says Mertens. “This has a much higher impact than just refactoring individual tests themselves.”

Avant’s end-to-end build time prior to CircleCI Enterprise was approximately 33 minutes (including environment setup and teardown). By using a custom Docker image to execute their builds in CircleCI Enterprise, Avant was able to optimize setup/teardown and ensure all their dependencies were pre-installed, reducing build times by about 8 minutes per build.

Avant saw an additional 5 minute improvement in build times because CircleCI maintains the application test environment through the full build.

Avant further reduced their build times to roughly 15 minutes end-to-end by increasing build parallelism to 30 containers per build with CircleCI Enterprise. Overall, Avant saved over 50% of their previous end-to-end build time from the performance improvements mentioned above.

Avant’s Mertens says the company has also realized other time savings for the development team by using API-triggered builds to automate tasks and perform early detection of test defects.

“One of the first ‘special’ jobs we used CircleCI Enterprise for was to automatically bisect test failures in our mainline branch when they occur,” says Mertens. “Intermittent test failures can cause any branch build to fail and have an adverse impact on the productivity of the entire development team. When our master build fails, if the failure has not already been bisected, we trigger an auto-bisect build on CircleCI. Previously bisect had to be done locally and could take anywhere from several minutes to an hour or longer to complete. Now we just let CI do it for us.”

“Another ‘special’ job we use CircleCI for is running ‘time travel builds’. In a time travel build, we execute our entire test suite in the context of some future date/time. This enables us to catch date-related failures before they occur. There is nothing worse than coming in Monday morning to a failing build, so we use time travel builds to detect test failures that would occur on a future date. This has had a very positive impact on our build pass rates and productivity which previously suffered when a test would fail across all branches just because the date rolled over.”

According to Mertens, “there was an immediate boost in developer happiness and productivity as a result of significant improvements to our end-to-end build times when we switched to CircleCI Enterprise.”

Transparency

Another benefit for Avant is that “CircleCI Enterprise is completely transparent at every step in the build process as well as run times for step-by-step and end-to-end build times. It has made reproducibility much more straightforward because it takes CI ‘magic’ out of the equation.”

Security

“CircleCI Enterprise has enabled us not only to ‘build better’, but also to move our CI infrastructure into a self-hosted VPC with AWS,” says Mertens. “This not only improves security because our code is hosted and tested on our own servers with restricted access to authorized users, but it affects cost as well.”

Cost

In addition to the security benefits that the behind-the-firewall version of CircleCI provided, Avant was also interested in the flexibility and potential cost-savings of running their own CI infrastructure. One key goal was creating a system that would provide the performance they needed at peak hours to avoid queued builds and the ensuing loss of productivity, without having a lot of machines sitting idle on nights or weekends.

About Avant

Avant is an online lending platform that is the leading provider of credit alternatives to middle income consumers in the US and UK. Avant offers access to unsecured personal loans ranging from $1,000 to $35,000, with funding as soon as the next business day, and has served more than 500,000 customers worldwide. Avant offers technology solutions to bank and non-bank partners via its Powered By Avant product to provide an innovative digital lending experience to their customers. Avant, founded in late 2012, has raised more than $600 million of equity capital and originated more than $3.6 billion in loans on the platform.