Two years ago, after rapid growth, our infrastructure’s efficiency was headed towards a local maximum, but we knew re-architecting was risky. We racked our brains, searching for a way to gradually update our platform, but we came up short. Eventually, we reached a daunting conclusion: we’d have to embark on a treacherous journey — a complete re-platforming of our infrastructure.
Because our tool is mission-critical for so many companies, we felt an added pressure to get our rebuild done quickly and to make sure it was done well. It was a nail-biting six months before the first customer saw any part of our new system. But, after careful planning, building, more planning, and rebuilding, we had finally emerged - and now had an updated platform that was far more performant than our original. When we made the new platform GA nine months after breaking ground, it confirmed our belief that our new system was smarter and more consistent, and that the re-architecting had been worth it.
Looking back, I can see 5 important concepts that were essential to our successful re-architecting. Here’s what I learned:
1. Before finding product-market fit, optimize for speed, not elegance.
There’s a saying about perfectly architected systems: we’ve never heard of them because those companies never get off the ground. When you first get going, picking the stack that works right now, and not over-optimizing for future possibilities, is the right choice.
Many startups focus on the wrong things before finding product-market fit. They waste time talking about the company they want to build, rather than creating a version one. Instead, focus all of your energy on building the product that will support the company that you want to scale in the future. Once you have that, you can figure out how to build a sustainable path for growth.
2. Fundamental architecture changes don’t happen incrementally.
After you’ve found your product-market fit, and you reach critical mass, waiting too long to refine your infrastructure can torpedo your reliability and ability to grow. At CircleCI, we felt this growing need to re-architect, but we also felt the immense danger in doing so.
Restructuring forced us to take a hard look at our principles: of all of the benefits of CI/CD, what could we abide in our rebuild? What would we have to toss out? How could we continue to make progress in other areas of the product while rewriting the core? We could have been stubborn and insisted on incremental improvements. But instead, we charged into this project because we knew what would be on the other side: better continuous delivery.
3. Constrain your scope.
Throughout the process of rebuilding, we did our best to keep our focus narrow. We focused on making changes that we knew would be impactful while minimizing changes elsewhere. We also exposed the new flow via branch-level configuration, so we could solicit real feedback without interrupting the day-to-day software delivery of our customers. This was important to us. We knew there would be a temporary impact on customers as we rebuilt, but we wanted to keep it as minimal as possible.
4. Customer feedback is key.
There were design flaws and architecture flaws and decisions that made sense on paper but didn’t work once we shipped them. Using customer feedback to help us identify those flaws was critical. The feedback helped us build a better, more intelligent, product, and we are so thankful that our customers gave us their input.
When you are in the startup stage, in the very early days, you need to understand whether you are doing something customers actually want and that the market will accept. Always take customer feedback seriously - and always aim to build a product that adds value to users and to the market.
5. Give your team a clear definition of “done.”
Admittedly, it was tempting to continue tinkering with the product forever, caught in a vortex of endless perfectionism. But we knew we needed to get back to our core philosophy: continuously delivering value to our customers. So we committed to this upgraded version as our default platform and have since built exciting features on top of it.
At the end of the day, some risks are necessary, and this was one of them. Even though re-architecting was an incredibly daunting task, it helped us improve the experience for our customers and pave the way for more product innovation. If you’re feeling like it’s time to rebuild, my ultimate advice is to listen - listen to your customers, listen to the market, and listen to the goals and boundaries that you set for yourself - use these tips to help set yourself up for success as you begin your rebuild.