Lately, I’ve been spending a lot of time thinking about the enormous growth in the adoption of DevOps within the tech space. DevOps adoption is changing the way that teams and organizations are building and releasing software. Now, most teams’ software development and release cycles have decreased from weeks, months, or even years, to hours and minutes. This is directly attributed to the adoption of continuous integration and continuous delivery (CI/CD) practices and principles.
CI/CD is a core component of modern-day software development. Teams are developing custom software development processes that fit their particular situations, and feedback mechanisms to provide insight into options for optimizing these processes. Teams are also developing tighter communication strategies that yield deeper collaboration and faster, more constructive feedback loops. Even the industry recognizes that CI/CD is critical to software development to the extent that it now has its own foundation, The Continuous Delivery Foundation (CDF). The following is the CDF’s mission:
The Continuous Delivery Foundation (CDF) serves as the vendor-neutral home of many of the fastest-growing projects for continuous integration/continuous delivery (CI/CD). It fosters vendor-neutral collaboration between the industry’s top developers, end users, and vendors to further CI/CD best practices and industry specifications. Its mission is to grow and sustain projects that are part of the broad and growing continuous delivery ecosystem.
With all of this growth in the CI/CD vertical, it’s obvious to me that individuals must develop CI/CD specific skill sets. If you’re not sure what I mean by CI/CD specific skills, that’s ok. I’m going to explain my thoughts on a new role that I’m dubbing CI/CD Engineer.
CI/CD Engineer
I’ll be the first to admit that a new buzzword-based job title and description is the last thing that this community needs, but I strongly believe that CI/CD engineers will materialize in the very near future as valuable assets to their teams. Maybe you’re thinking:
- What are some of the characteristics that they possess?
- What duties would they perform that would warrant a separate engineering role?
Read more: The Path to Platform Engineering
Before writing this post, I’ve discussed this concept with various members of the community ranging anywhere from developers, SREs, QA Engineers, and scrum masters. The conversations always started with me asking this question: How would you describe a CI/CD Engineer and their associated duties?
Some of the more common responses were:
- DevOps Engineers don’t need assistants.
- Sounds like a Release Manager to me.
- I don’t see a difference between a DevOps Engineer and CI/CD Engineer.
- CI/CD Engineers would be responsible for leading the charge on CI/CD and keeping teams motivated.
- An individual directly responsible for the health and operation of CI/CD tools and platforms within an organization.
- A resource that is knowledgeable in optimizing their team’s development and release practices.
I compared the responses I collected against my ideas and the critical elements of such a role became more apparent.
The characteristics of the role pertain to skills that a candidate should possess. The duties of the role define the things they would be responsible for. Breaking these elements out in this manner helped me focus on my description. First, I’ll discuss the characteristics of a candidate for this role. Then, I’ll move on to discuss my thoughts regarding a CI/CD engineer’s duties.
Characteristics of CI/CD engineer
The following is a list of five skills that I believe an ideal CI/CD engineering candidate should possess:
- Strong communication skills
- Keen analytical skills
- Ability to decompose complex processes into understandable components
- Proficiency in automating and optimizing processes
- Competent in team building and team communication strategies
This list is not exhaustive, but it feels like a great starting point. Now, I’ll offer up a more detailed explanation for each of these characteristics.
Strong communication skills
This role will definitely converge on multiple verticals which will require individuals to competently communicate and interact with various teams. The ability to clearly communicate is critical for this role.
Keen analytical skills
Much of the work in this role will require an individual to gain a deep understanding of a variety of concepts and how they relate to and impact the relevant domains. Being able to accurately assess respective landscapes is a very important step in improving inefficient processes.
Ability to decompose complex processes into understandable components
Breaking down complex topics into digestible components is a great trait to possess in many roles, but it’s especially important in this context. Combined with keen analytical skills and having the ability to explain complex concepts, teams can understand and capture their current operational states along with any existing deficiencies. Having an accurate view of operational landscapes enables innovative solutions for achieving and maintaining desired states.
Proficiency in automating and optimizing processes
Identifying deficient steps enables teams to expose pain points and bottlenecks that diminish the effectiveness of processes and to address them with viable solutions. For instance, imagine a process that executes sequentially, meaning one step needs to complete before another can begin. This behavior is often called blocking. It’s my experience that most sequential processes have steps that can be executed in parallel which removes this blocking. Being able to devise and coordinate these optimizations is especially beneficial in this role.
Competent in team building and communication strategies
This characteristic is based on a collective of the traits I’ve already mentioned with the addition of developing and maintaining credibility within teams and organizations. Credibility is the quality of being trusted and believed in. People are way more comfortable buying into and supporting decisions when they’re confident in the individuals championing the efforts. The goal here is to develop consensus around decisions and to execute plans that benefit all of the stakeholders.
Duties of a CI/CD engineer
Here I’ll share my ideas on the top five duties that a CI/CD engineer would perform:
- Develop CI/CD principles
- Review and modify CI/CD principles, iteratively
- Maintain CI/CD tools/platforms (if applicable)
- Develop and maintain pipeline configurations
- Automate processes
Develop CI/CD principles
Adoption of CI/CD principles produces huge gains in efficiencies. These practices are composed of processes that are defined by stakeholders and facilitate optimized software development. A CI/CD engineer will be an authority on everything “CI/CD”. As an authority, they will be actively involved in the development and implementation of these related processes.
Review and modify CI/CD principles, iteratively
Once practices are established, it’s critical that teams continually revisit them to verify that they support current operations optimally. CI/CD engineers will be in a unique position to develop a deep understanding of how respective teams are operating and interacting with the other units. They’ll have insight into what’s working and what’s not with the added ability to determine and offer corrective actions. Iterative reviews will provide consistent visibility into the processes that drive software development and surface options for improvement.
Manage CI/CD tools/platforms (if applicable)
The next step after establishing and adopting CI/CD practices and principles is leveraging CI/CD tooling and platforms that will facilitate the execution of related processes. CI/CD platforms bring our processes to life and provide the automation that executes specified pipelines. Implementing and managing CI/CD tooling requires a combination of skills, DevOps for sure, but I suggest that in this aspect a bit more emphasis on Ops is appropriate. I say this because ensuring that CI/CD tooling operates consistently definitely falls under the “operations” wheelhouse. This particular duty is also dependant on architecture and operational landscapes.
In my experience, organizations operate their CI/CD architectures in 2 different modes:
Managed CI/CD:
The CI/CD architecture is provided by a 3rd party vendor such as CircleCI. These services provision, manage, and scale the underlying CI/CD infrastructure with very little effort from technical teams. These services are designed to directly serve developers and strip away all of the hard parts of CI/CD. Engineering teams are empowered by these tools to develop software and not to be bothered with the resource-intensive and time-consuming management of CI/CD infrastructure.
Self-hosted/on-prem CI/CD:
In my opinion, Managed CI/CD should be on every organization’s radar, but due to circumstances such as data laws and government regulations, self-hosted platforms are still required by certain organizations. Self-hosted CI/CD requires teams to provision and manage all aspects of their CI/CD infrastructure. Teams manage these infrastructures holistically which means they have to design, implement, and manage connectivity, servers/nodes, operating systems, and the chosen CI/CD tooling that must run securely on this elaborate infrastructure. This requires lots of time and effort on behalf of DevOps teams and is a more complex operation.
Regardless of CI/CD architecture, achieving efficiencies in CI/CD is critical. Once the tooling is operational, it must remain that way or risk a decline in performance. Of course, there are many CI/CD tools available, and selecting the most appropriate tooling is no small feat. I advise vetting multiple solutions and choosing the tooling that best serves your team’s needs.
CI/CD engineers could be responsible for some or all aspects of these CI/CD infrastructures. Ideally, this is a shared duty between CI/CD engineers and DevOps Engineers. CI/CD engineers manage the CI/CD tooling services and DevOps manages the underlying architectures. At the very least, the CI/CD engineer must have deep knowledge of the CI/CD platforms they’re supporting including the underlying host infrastructure.
Develop and maintain pipeline configurations
All CI/CD tooling requires a form of pipeline configuration which is the mechanism that specifies the various steps and segments to execute within your CI/CD processes. Configuring pipelines is completed in different ways depending on the CI/CD tool being used. Some tools make use of a Graphical User Interface (GUI) to configure pipelines, where others require that pipelines are specified in code. In all cases, CI/CD pipelines must be specified and maintained, and this duty will fall squarely on the CI/CD engineer.
Developing, managing, and executing CI/CD pipelines is the biggest responsibility that this role owns. The pipeline configurations orchestrate the execution of the steps specified. In other words, pipeline configurations are how we program the CI/CD tools to do what we want.
CI/CD engineers will ensure that pipelines are properly defined and performing optimally. They will have a deep understanding of the pipeline’s goals and the transactions occurring within them. They will interface with DevOps teams to coordinate and optimize the steps that execute within the pipeline. The CI/CD engineer will document the pipelines so that all of the interested parties are aware of how their software is built, tested, secured, and deployed.
Automate processes
Automation is at the core of CI/CD. It facilitates processing pipeline commands, inter-platform connectivity, and valuable integrations with 3rd party services. If you peel back the layers, at the core, CI/CD engineers will design and manage the automation that powers continuous integration and delivery operations. The level of experience can vary, but a foundational knowledge of automation will be required to be an effective CI/CD engineer.
Technical skills of a CI/CD engineer
Finally, below is a list of the technical skills that a CI/CD engineer will possess:
- Script writing (languages are irrelevant)
- Ability to interpret and write source code (languages are irrelevant)
- Management of infrastructure assets (networks, servers, operating systems, DBs)
- Familiarity with software packing tools (.exe, .deb, .rpm, Docker)
- Familiarity with version control tools (Git, Subversion, Mercurial)
- Management of cloud providers (AWS, GCP, Azure)
- Familiarity with security/vulnerability tools
- Familiarity with code coverage analytical tools
- Familiarity with monitoring tools
The skills I mentioned above are not immediately necessary as long as the plan is to achieve and grow missing skills while occupying the role.
Conclusion
In closing, I am laying out a case for a new role I’ve named CI/CD engineer. I’ve described some characteristics and duties that I believe are directly tied to the role. These characteristics and duties are ideas that would be essential if the role were to exist.
When I combine the character traits and duties that I’ve suggested, I recognize that what I’m proposing is a new role composed of bits and pieces from multiple existing roles and that feels right to me. CI/CD engineers should be a little bit developer, operator/SRE, release manager, scrum master, cloud infrastructure architect, business analyst, systems integrator, security officer, and network engineer. This is a big ask, but individuals with solid experience and/or knowledge in some or all of the roles mentioned will make great CI/CD engineers.
Again, these are my thoughts on what constitutes a CI/CD engineer. I would love to know your thoughts and opinions on the role so please join the discussion by tweeting to me @punkdata. Hopefully, we can develop this idea into existence together.
Thanks for reading!
