A brownout is an intentional but temporary disruption of service. Why would a company purposely cause a brownout and when does it make sense? In this post I will explain the what and why of brownouts, and provide some recent examples.
What is a brownout?
The term brownout comes from the utility industry. For electric companies and their customers, a blackout (or total loss of power) can be devastating. Instead of letting things get to the point of total disruption, workers at the utility watch for signs that a blackout is imminent. The utility can then reduce voltage for everyone, or disconnect service in some areas for a limited time. The effects are much different that those of a blackout. For example, the lighting in an older house with low voltage might dim, with the color fading from its normal yellow, skewing brown. The idea was that the ramifications of this “brownout” would be less severe than a blackout that would affect everyone.
The tech industry has adopted the brownout concept and applied it to servers and the services that run atop servers. A tech company might selectively limit access to their services if hardware failed or if an influx of traffic threatens to bring their entire network down. Tech companies have found that brownouts can ease the disruption for their customers when a service or feature must be phased out (also know as “sunsetting”).
Why implement a brownout?
A well-planned and intentional brownout can be a vital tool to alert users to critical changes, collect data, and prevent future loss of service. Here are some of the things a SaaS provider like CircleCI might consider when planning a brownout:
- Demonstrating the scope of the impact
- Understanding the impact on indirect users
- Getting customers’ attention so they can manage the change
Demonstrating the scope of the impact
In February 2022 CircleCI announced the deprecation of our Ubuntu 14.04 and 16.04 images. These images will be deleted on May 31, 2022. Here’s a hypothetical scenario to consider:
SavvyTech, a CircleCI customer, came across our blog post and saw the headline. They looked through their projects’ configs and didn’t see any of these deprecated images. Great! They might be thinking “that won’t affect us, we’re good.”
Fast forward to the end of March when the first brownout for these deprecated images occurs. SavvyTech pipelines for some projects fail for 10 minutes due to missing Linux images.
- What happened here?
- Why were they affected?
It turns out that they were using an old version of a CircleCI orb that was using one of the deprecated images. This doesn’t show up in their config but clearly still affects them. Once the 10 minutes are up and service is restored, they make changes to start testing out an updated orb in parallel to their regular work.
The brownout in this scenario demonstrated to SavvyTech the full impact of the image deprecation. In fact, the impact is larger than they first anticipated. The brownout caused 10 minutes of downtime, not the hours or even days that could have happened if they didn’t realize until later that the images had been permanently deleted.
Understanding the impact on indirect users
Technology connects us in many ways. For tech companies that means if AWS goes down, especially us-east-1, we all tend to feel it whether we’re an AWS customer or not. The trickle-down effect can impact Slack, Facebook, or even Spotify, which can affect everyone from development teams to casual social media users.
This phenomenon makes brownouts an effective way to alert users who aren’t direct customers of a company.
Here’s an example. In 2021 JFrog decided to sunset their BinTray hosting service. They announced the plan on their blog, sent emails to their customers, among other communications strategies. What JFrog didn’t do was reach out to a developer in Germany (I will call her Molly) who runs an older, custom fork of Homebrew on her MacBook Pro. Why would they? How would they identify her? JFrog also ran a brownout on BinTray.
Molly isn’t a customer, she doesn’t even know what JFrog is, but she’s still affected by BinTray being sunsetted. Molly finds out about the change when she tries to brew install and gets a error about a BinTray URL. Turns out, Homebrew used BinTray to store bottles (binaries). The BinTray brownout that JFrog ran alerted Molly, along with thousands of others, to the upcoming sunsetting of BinTray. Molly reaches out to the Homebrew community and learns that she needs to upgrade her fork of Homebrew to a more modern version, and when that needs to happen.
Getting customers’ attention
At its core, a brownout is a non-destructive tool for getting someone’s attention. Sometimes emails aren’t delivered or stay unopened, blog posts don’t get read, and in-app banners are ignored. In the DevTools space, you’re trying to reach developers who spend the majority of their time in a terminal or VS Code, so how do you reach them? If you’re lucky, there might be a CLI you can use. But sometimes you don’t even have that.
Here’s a specific space I’m familiar with: continuous integration. Often, a dev isn’t looking at anything other than their editor and the GitHub PR page unless there is a failure.
They might hop over to the web app to see what test or download failed. A failure caused by a brownout might be the only way to reach out to a heads-down developer who’s simply knocking out Jira tickets one after another. This dev could be so keen on achieving Inbox Zero that a sunsetting email gets deleted before they ever see it.
When it comes to deprecating important or widely-used features, attention grabbing is key.
Assessing the risk of the brownout
It’s definitely stressful when users have their service disrupted. It can cause frowns all the way around, especially for the support team. But what is the alternative?
Disrupting a user’s service for 10 minutes or an hour is not ideal. But having that user’s service shut off indefinitely by the EOL date is much, much worse. Sunsetting a service or feature is often irreversible, and the required action by the user sometimes requires time. Getting customers’ attention before service is permanently removed gives them time to create a plan and then successfully execute that plan.
Real-life examples of tech brownouts
Here are examples of some recent brownouts from tech companies. Even from this small group of examples, it’s that clear brownout length and frequency is still being tested and refined.
CircleCI - We’ve deprecated the very old and insecure Ubuntu 14.04 and 16.04 releases. To alert users to these changes, we scheduled brownouts in March, April, and May of 2022. Information about the images, the timeline, and more can be found in our deprecation announcement post.
GitHub - In September 2021 GitHub announced that they were deprecating old and insecure OpenSSH ciphers. They proceeded with a few brownouts of those ciphers before ultimately discontinuing them in March 2022.
JFrog - As mentioned earlier in this post, JFrog deprecated their hosting service BinTray early in 2021. This led to several brownouts of the service before it was shut down for good in May 2021.
Codecov - After some trial and error, Codecov came out with a new and improved Bash uploader for their service. This meant deprecating (and browning out) the older Bash uploader. They explain the reasoning and provide details for the plan in this blog post.
PyPI - PyPI isn’t a company like others on this list, so there is no central place to point people to for details. Nonetheless, PyPI, an open-source project and an important piece of the Python ecosystem, ran their own deprecation and brownout. This was communicated in many places, including a mailing list and several community member blogs.
Conclusion
While brownouts are not the only available tool for feature deprecation, they are effective. Brownouts can be especially helpful later on in the deprecation process. In an age of notification overload, they are a way to gain the attention and awareness of not only your own customers but integrators and downstream users as well.
As I have mentioned several times in this post, CircleCI has deprecated old and unsecure Ubuntu images recently. Brownouts for these images are scheduled for March 29, April 26, and May 19, 2022.
You can get more information about this image deprecation and find the complete schedule for the brownouts in this blog post. The post contains links to these migration guides to help you with the move to using the updated images:
You can use these resources to discover the scope of the impact of the image deprecations for your team, understand the impact, and manage the change to the new, more secure images.
