Testing With Untrustworthy Data Means Nothing: how CircleCI came to grips with declaring bankruptcy on analytics and rebuilt from the ground up
When I joined CircleCI as a Growth Engineer, I came on board with one mission: improve conversion rates. Having freshly transitioned from an ecommerce startup, I had spent the past six months designing and iterating on funnels which turned user acquisitions into conversions. During that time I had learned a pretty basic formula for improving conversion rates:
- Enumerate the key funnels
- Find an area of weakness in a funnel
- Hypothesize why users are falling off
- Test that hypothesis
- Look at the data to either confirm or deny the hypothesis
- Lather, rinse, repeat
So when I joined CircleCI Rishi (our Growth Project Manager) and I went to work. First we enumerated our key funnels: non-user to user, user to paying customer, and then paying customer to higher paying customer. We used the knowledge and data we had at our disposal to identify what funnels we had and which were underperforming. And last, we launched test after test to try and improve acquisitions, conversions, and upgrades.
But as we started to try and measure the success of these tests, we kept running into the same problems. We either couldn’t trust the data, or we were missing the data we needed to tie the test to the larger picture: the health of the business. After careful consideration, we decided it was time to declare bankruptcy on a broken analytics implementation and build a new one from the ground up.