DevOps isn’t dead: How platform engineering enables DevOps at scale

As the popularity of DevOps has grown, so has the complexity of software application development. Developers must learn new tools and maintain infrastructure while coding and prioritizing ops tasks alongside feature development. The weight of these demands not only leads to lower levels of productivity and effectiveness, but it also shows up in stress, burnout, and reduced job satisfaction.

Platform engineering is a logical extension of a mature DevOps practice, designed to work alongside existing DevOps concepts while alleviating the associated cognitive load. The general premise is to bring self-service to DevOps. Platform engineers abstract the complexity of common DevOps processes into an internal developer platform (IDP) that provides a standard set of tools for building and deploying applications.

This article will review the challenges of doing DevOps at scale and why DevOps practitioners are looking to platform engineering to solve these core issues.

Enhancing DevOps with platform engineering

Platform engineering aims to enhance DevOps, not replace it. Platform engineering removes the need for developers to learn, develop, and maintain infrastructure tools and processes. Some common examples include using Helm to manage Kubernetes clusters or writing Terraform code to deploy infrastructure automatically. Instead, these processes are abstracted into an IDP so developers can focus on building software.

The cognitive load of DevOps

Although DevOps was designed to make software development more efficient, DevOps engineers often face numerous demanding tasks that increase their cognitive workload. Let’s walk through a few common examples.

Process design and maintenance

DevOps engineers often need to design new processes for new software projects, such as the continuous integration and continuous delivery (CI/CD) process. DevOps engineers may also need to spin up the infrastructure for development environments while ensuring compatibility with production environments. This involves managing Docker files, Helm charts, and Terraform code, which require periodic maintenance and support as projects evolve. CI/CD pipelines also need building, and although engineers can take some parts from previous projects, the new project’s tests or build requirements add an extra layer of complexity.

Existing processes must be scaled up or down to meet current demand. These processes include scaling infrastructure to meet new processing or storage requirements and modifying existing workflows designed for a small team of engineers that has since grown.

Tech stack management

DevOps practitioners also manage ownership of many aspects of the software engineering lifecycle. This includes managing third-party tools and products on top of internal codebases and infrastructure. Other developers will also require code reviews and meetings to discuss potential solution options that require specialized DevOps knowledge.

Monitoring and reporting

Finally, DevOps engineers must ensure systems are logging correctly and that metrics can be collected and analyzed. Keeping on top of different software applications’ performances is vital to ensuring they all run smoothly. It also helps engineers understand potential areas that require scaling.

Beware DevOps burnout

All these responsibilities, on top of meeting service-level agreements (SLAs) and other internal business goals, cause a massive cognitive strain on DevOps practitioners. Each task and process creates extra overhead that DevOps practitioners must deal with, often removing resources from their primary goal of innovation and optimization.

As the cognitive load increases, so does associated complexity and stress, causing burnout, mistakes, and entropy to build. This significantly decreases team productivity and can ultimately stifle innovation.

So, how can platform engineering reduce the load and enable engineers to achieve their primary goals?

What do platform engineers do?

Platform engineers develop IDPs that solve everyday DevOps tasks, reducing or removing overheads entirely. This allows DevOps to focus on tasks that provide real business value more quickly.

Automate DevOps tasks

The main goal of IDPs is to provide a standard and reusable approach for all teams. Setting up a repository, assigning the correct team members, spinning up an environment, and configuring a deployment pipeline are already built into IDP tools and services. These can be broken down into four key management areas: infrastructure, deployment, configuration, and user access control.

Whether using an off-the-shelf IDP or developing your own, most IDPs offer a user interface (some are command-line only) for developers. Typically, an API layer requests the different services to perform an automated action. This action can be to create a git repository, spin up a database using a terraform script, or any other common DevOps task. Developers then use the IDP to initialize the core parts of a new project automatically.

Abstracting away the lower-level details of performing these actions prevents developers from diverging in their approach. Infrastructure is configured and deployed in a standard way, making maintenance simpler.

Support developer autonomy

IDPs also promote developer autonomy through self-service tools. This prevents lengthy lead times in deploying project resources caused by designing and building the infrastructure from scratch. Now, developers can use the IDP to do the work for them. IDPs also remove the need to create tickets in the backlog for setting up new environments and waiting for them to be prioritized and completed. Instead, developers can perform these actions via the IDP interface.

Enforce security by design

Finally, IDPs can be secure and compliant from the start. This means that platform engineers bake in security and compliance features into IDP services so that infrastructure follows security best practices when it is deployed. They can also ensure that Git pipelines are configured to automatically scan code for vulnerabilities before being built and released.

By making secure and compliant processes integral to development workflows, platform teams elevate DevOps pipelines into robust DevSecOps pipelines, ensuring that security is a foundational element throughout the software development lifecycle. This approach benefits organizations by reducing both the burden on developers and the risk of data breaches.

Incorporating platform engineering into DevOps

Let’s examine a hypothetical use case of how platform engineering methods could improve a DevOps team’s workflow.

The manual approach

Imagine a business that designs web applications that each follow a similar architectural pattern. There’s a database, an API, and a web-based front end. There are some pre-built Docker images and CI/CD templates.

Yet, everything is done manually. DevOps engineers must create Docker files for every project, implement Terraform scripts, and build Git pipelines. Then, they will work with developers to ensure the environments meet their needs and add monitoring and alerts to production infrastructure. These tasks occur in conjunction with responding to SLAs for existing infrastructure and performing regular maintenance.

This creates massive stress for DevOps engineers. The main issue is that all tasks are directed to the DevOps team, regardless of their complexity. Consequently, work stacks ultimately increase the lead times for developers looking to progress their projects.

The automated approach with platform engineering

To alleviate these issues, a platform team could automate almost all this work by building it into an IDP.

For example, instead of manually setting up Git repositories, developers could request a repository from the IDP, which would then create it. The IDP would then assign the right user group and automatically integrate the correct CI/CD template.

The same pattern applies to creating development environments and deploying core infrastructure. The IDP acts as a self-service platform for developers to request services and apply configurations, knowing security best practices and monitoring are built in by default. IDPs can also automatically set up projects in project tracking software and documentation templates.

Platform teams optimize DevOps practices

As you can see, platform engineers don’t replace DevOps processes. They enhance them by building a set of standardized patterns into a self-service internal development platform. This removes the burden of project initialization so teams can start providing business value immediately, rather than spending the first few weeks of a project setting up and working through teething issues.

Furthermore, because developers will be more autonomous, platform engineers can then concentrate on solving bigger architectural challenges and feeding that back into the IDP. In this way, they can improve existing services and strengthen newer systems going forward.

Conclusion

DevOps grew in popularity as it provided developers with the freedom to manage the end-to-end lifecycle of the application. However, this great power came with great responsibility. Over time, it increased the cognitive load required to manage and operate these processes at scale.

Because of this, platform engineering is something every organization with DevOps teams should consider. Consolidating core operations processes into an IDP drastically reduces the inherent cognitive load required by DevOps practitioners. Turning common DevOps processes into a set of self-service APIs also reduces project lead time, so developers can provide real business value swiftly.

If you are ready to start scaling up your DevOps practices with platform engineering, start by signing up for a free CircleCI account to see how best-in-class continuous integration can help you automate critical processes for faster, more consistent delivery outcomes.