What is growth engineering?
Growth engineering is a practice in which product, engineering, and design support a company’s growth efforts from within the product itself.
Growth engineering has gained traction in consumer-facing companies. This practice has gained plenty of traction in the SaaS world over the last decade, to help support growth of self-serve users who often purchase services without the involvement of traditional sales teams.
Growth engineering at CircleCI
CircleCI has recently created a set of growth engineering teams to involve engineers more closely in increasing product adoption. Our growth team’s goal is to optimize our users’ experience to make them as effective with our tool as possible.
But, as a B2B DevOps tool company, our customers are engineering organizations, not individual users. We therefore face some unique challenges that aren’t commonly seen by growth engineering teams in B2C organizations. Needing to measure the impact of our experiments on paying organizations, and not users changes the way we approach growth engineering.
A different measure of success
When companies talk about growth, especially those companies whose audience is consumers, one of their goals is usually to keep users engaged with their product. For example, with a product such as LinkedIn, the goal is to keep visitors engaged with the platform by visiting the platform often, and when on the website, to stay on the site as long as possible. Finally, they want visitors to come back on the website as often as possible.
Our situation is quite different. As a DevOps tool, CircleCI is mission-critical and allows our customers’ teams to deliver their code to millions of users. Our daily touchpoints with the users of our platform are fairly light, and that is by design. Our tool normally operates in the background, and users do not make significant changes to their CI systems on a daily basis. Therefore, time spent on our platform isn’t a good proxy for user success the way it would be for a social platform.
In essence, from a CircleCI user’s perspective, our daily UI is a simple integration point that will inform them if their CI/CD pipeline was successful or not, and simply looks like this:
The use cases where our users need to go in our system are fairly limited: updating their configuration, troubleshooting a build failure, or setting up a new project, for example. While these are performed less frequently, they often have a sense of urgency around them. That means that when users do come in our system, our success is measured when they spend less time in our system, not more.
With that goal in mind, most of our experiments are targeted toward finding ways to make our UI more effective for our users by:
- Improving error prevention; catching errors before they happen.
- Reducing the time required to resolve the errors.
- Enabling engineers to collaborate on configuration or errors.
- Providing contextual information so engineers can make changes faster.
- Shortening feedback loops when making changes to the configuration.
- Improving the chance of success for CI/CD pipelines.
- Reducing the time required to build and deploy.
This brings us important challenges we face when experimenting at CircleCI.
Challenges measuring growth experiments
Our users aren’t our paying customers, and our paying customers aren’t our users
At CircleCI, anyone can start building for free today, and thousands of engineers on our platform use CircleCI in bootcamps, on side projects, OSS projects, or to build startup prototypes for free. Our paying customers are almost exclusively professional organizations, as they need more capacity and need access to advanced features only available on paid plans.
Organizations are defined as a collection of multiple engineering teams, each composed of one or more engineers. Typically organizations map to a company, but in theory, a company could have multiple organizations defined in their source control management system. The key concept for us is that we do not convert users from unpaid to paid: we convert organizations. Organizations, not end users, pay for CircleCI’s offerings.
With that in mind, we need to gear our experiments towards users that are part of organizations that have the potential to convert, and to measure the impact of our experiments on organizations, not users. There are some important metrics that we use to help us define an organization’s likelihood to convert, such as:
- The number of members in the organization.
- If they pay for their source control system.
- What they pay for their source control plan.
Audience constraints
First, the audience of growth engineering teams is constrained to a subset of our users that do have the potential to become paying users. This constraint means we often face more difficulty reaching statistically significant results for our metrics. In some cases, statistical significance is never reached because the audience is too restricted for the experiment. For example, if we target the administrators of new organizations only, the number of people seeing those experiments could be as low as hundred on any given day. There’s no easy solution to this - we have to manage and prioritize our experiments accordingly to how likely they are to become statistically significant, or sometimes, just accept that we will not get results for weeks.
Org-level experimentation
Another challenge we face is that most of the industry’s off-the-shelf experiment tooling is geared towards user-level experimentation and metrics. While there is some value for us in this data, the available tooling doesn’t allow us to group the data at a higher level. To properly understand how organizations use our product, we need to be able to measure the impact of our experiments at the organizational level. We also want consistency of experiences presented to all users within a single organization. This is critical for us: in order to truly understand the impact that our experiments have on an organization, we need all users within that organization to have the same experiments and measure their impact at the organization level. At CircleCI, we use Optimizely, which lets us deliver experiments by organization and funnel the data for analysis at the organizational level.
Conclusion
Growth engineering is gaining momentum in our industry as a product development approach to support business growth. However, it is still a new practice, and most of the adoption has been from companies that cater their growth at the user level, and want to keep users actively engaged with their product as much as possible.
As we’ve demonstrated, we’re facing unique challenges with this methodology due to the type of tooling and users we have. We’re focusing on keeping our users engaged, but for us that means getting users to spend less time on our frontend than more. We’re also building our own metrics and data pipeline to ensure that we can measure the impact of our experiments at the organizational level, rather than than at the user level.
Want to contribute to growth engineering at CircleCI? See our open roles.
