What I learned from developing a GitLab support feature for CircleCI

Earlier this year, CircleCI added GitLab as the third version control system that we support, in addition to GitHub and Bitbucket. At CircleCI, it’s vital that we meet our users where they are, and many of our users are on GitLab. We were happy to make it possible for our users to build, test, and deploy via the GitLab platform.

As a staff software engineer on the onboarding experience team, I want to tell you about what it was like to develop this experience — the challenges we encountered, how we overcame them, and what we learned that we’ll carry forward.

The silent launch period

My team chose to silently launch GitLab availability so we could gather early feedback and make improvements before releasing the experience to all users. We agreed that the minimum functionality required before releasing, even silently, meant that a user had to be able to go from not having a CircleCI account at all, to successfully creating a new project connected to a GitLab repository. The experience had to contain everything a user would need to complete their journey with no manual interventions.

By silently letting CircleCI users discover this new functionality, we were able to capture users who were already actively interested in trying out the GitLab experience. Our theory was that these users would be motivated to go through the entire experience, even if it was a little bumpy, and we would be able to get valuable information from them.

Because users were discovering the experience on their own and we didn’t have direct interaction with them, we had to make sure our monitoring and analytics were securely in place so we could see which parts of our system were slowing users down or preventing them from being successful.

Getting access to the new feature: Personal Access Tokens vs OAuth

We offered 2 options for users to set up projects: “personal access tokens” or via the passwordless access application, OAuth. We quickly found that users who tried to use personal access tokens often didn’t understand how to create the tokens, what permissions they needed, or what they were used for. Once we noticed that access token users were having trouble, we added more text and links to documentation, and we tried linking them directly to the GitLab tokens page.

Ultimately, we found that personal access tokens were too cumbersome for our users, so we decide to do away with them altogether in favor of OAuth. Now we see many more GitLab users successfully making it through the sign-up process.

Continuous delivery was critical during this phase. Any time we noticed that users were struggling to understand something, we immediately began working on a fix. Once we shipped a fix and merged it, it was live for all users within 20 minutes. This left my team feeling empowered as developers because we were able to deliver value as often as possible and we saw the experience improve in real time.

General availability: learning from feedback

The General Availability launch was all about gathering more robust feedback. With increased traffic to GitLab support, we were better able to identify trends and observe common areas where users were getting stuck or having a poor experience.

We largely relied on our built-in analytics and monitoring capabilities to inform how users were progressing through the experience. If there was anything we could ship quickly to improve the experience, we prioritized those fixes and aimed to deliver them as soon as possible. Any common problem areas or places where users got stuck were grouped together and we would brainstorm larger initiatives to improve these areas later.

During this period, we again saw that many users were getting stuck in the sign-up process. The way our GitLab support works is that a GitLab account can only ever be linked to one CircleCI account. We found that when users were testing the experience, they would sometimes use a temporary GitLab account before connecting their real one.

Then, when they tried to use their real GitLab account, their permission would be denied. We didn’t do a good job of communicating that problem to users — we actually failed to show an error message at all so it looked like nothing was happening, which users found very frustrating.

Once we realized this, we were able to quickly deliver a change to distinguish this error case specifically and provide an actual error state for users, giving them more clarity into the issue.

After implementing these changes, we would observe the impact daily. Overall, we found that more users were successfully accessing the experience after we had shipped improvements, which was really gratifying to see.

Delivering value quickly

Throughout this experience, my team learned how crucial analytics are to a project like this and that they should be put in place with the same attention we give to the rest of the project’s design. Many of the early changes we made to the experience were around adding tracking capabilities. If we can be more diligent about planning our analytics capabilities at the start, we won’t have to play catch-up next time.

We also learned that silently launching was a useful tool. When we’re developing, we tend to be very focused on our specific experience and it can be tough to look beyond that. By silently launching early, we had more time where the experience was live and we were able to gather real feedback on what we built. Having the space to step back and look at the whole experience gave us a chance to find rough edges and address them before more customers discovered them.

One of the most satisfying things about working on this project is that my team had a lot of control over the experience. We were able to deliver value to our users quickly and effectively. Our testing practices and controls meant we could ship our changes with confidence. We were in a position to ship multiple changes to production daily, which helped us learn more about our own system and let us feel in control of this project from beginning to end.