It’s the middle of the workday and I’m at my computer, running my tests locally to make sure they don’t break. I’m working on a project I’ve never worked with before on a team filled with people I look up to. Imposter syndrome is hitting me especially hard today. I’m worried about making the wrong impression because I forgot something trivial in my code. Running my tests locally means no one else needs to see my mistakes and that gives me peace of mind. So I rerun them multiple times just to make sure my code is airtight.
I push up my code, open up a pull request, and wait for tests to pass remotely. To my complete surprise, I am greeted with a big red circle. Not only do I see this failure, but the rest of my team starts to see it too as they begin reviewing my code. I panic. Embarrassed, I begin frantically trying to fix the build before more souls begin to notice. At some point, I feel the “hat of broken builds” fall upon my head. I work furiously until I am able to make builds pass again and rid myself of this devastating mark of shame.
Becoming friends with your broken builds
Working with continuous integration (CI) systems can sometimes make you feel like you are being publicly shamed for trying your best to just do your job, but a few mental shifts can turn them into an invaluable development tool instead. After many years of agony as a newbie dev, it become clear to me that failures were actually good and useful, and I began to look at CI and my tests completely differently.
So, your build fails… what exactly does that mean? Checking the job output should point you to a failing test. This could be a test that you expected to fail because it directly relates to the code you are changing. It also could be a test you didn’t expect to fail because at first glance it seems unrelated to your change–but upon closer inspection you realize that the code you wrote actually broke something in some far-off land you never considered. That is great information to have and it saved you from deploying something that would break the application or service for users.
CI is ultimately trying to help you be a better engineer by forcing you to look at problems that you may have never considered during the development process. These failures are not really failures at all, but are instead opportunities for learning! When you learn how to fully use your CI pipelines, not only do you allow yourself to work better but you also start working smarter.
Changing my testing strategy
I used to run all of my tests locally, every time. Now, I run only a subset of tests that are related to the change I am making. This subset of tests takes less than a second to actually run and gives me immediate feedback on my updates. Reducing the size of my test suite allows me to shorten the feedback loop such that I can run it after every little change. The more immediate my feedback is, the quicker I can produce quality work, even if it means that I don’t run all of the tests immediately. When I get to the point where I feel good about my code, I push it up to my PR where it will be tested again, but this time more thoroughly and in a consistent environment with everyone else’s code. If that fails, my colleagues won’t care - it’s only in my branch. When I started treating CI as a remote computer, it drastically increased my output – leaving me with quite the opposite feeling of the dreaded shame machine.
When I started treating CI as a remote computer, it drastically increased my output.
Ideally the tests that you should be running are the tests that are the most likely to fail. For instance, when I’m testing locally I’ll know what functions I’m changing and will run the tests for those functions. They are the most likely to fail as a direct result to my changes. As a result I save myself the time and decrease my feedback loop for that feature. When I’m confident that my tests pass locally, I run the rest of the suite in CI. The rest of the tests should be tests that have the potential to fail because that is the only way they can give you the information you need to continue to make a great product. If your tests are testing things which are likely to never ever fail, those tests end up wasting time which could be used to test more important things. What’s more, these are tests which should be either deleted, or better yet, made more useful by allowing them to test less tautological scenarios.
Getting a red build isn’t a failure. But there are certainly scenarios in CI which are real failures, and which are worth resolving quickly and avoiding in future. For example, if you are sharing a branch with anyone else, it is important to make sure that you do not prevent them from being able to merge into that shared branch by accidentally introducing failures there. Those failures can be introduced into your own branch and tested in isolation before merging into a shared branch like
master. I’d also consider any test which does not provide meaningful information to you a failure. If your tests are always prone to failing due to some non-deterministic or flaky behavior, these tests will ultimately be ignored because the signal-to-noise ratio is far too low to be useful–and they may even cause you to doubt genuine failures elsewhere. These tests should be considered for deletion or refactoring in order to be made more useful. Likewise, any test that passes all the time and never fails will likely be overlooked and isn’t providing any useful information. The sweet spot is writing tests in such a way that they fail just enough of the time–imparting useful, actionable information that will help you improve your code.
I used to dread seeing that red dot next to my build. The more that I think about how much help that humble icon has given me in my career, the more I’m able to look past that dread and realize that my broken builds are actually friends, not foes.