Unlocking efficiency with CircleCI’s new multi-repo project model
Senior Solutions Engineer
In a traditional CI/CD pipeline, the instructions for how your pipeline should run are stored in the same repository as your application code, creating a rigid 1:1 relationship between your pipeline and project code.
This approach has benefits, like keeping pipeline logic close to the code it supports, but it also introduces challenges, especially for organizations managing multiple repositories or microservices. Every change to a pipeline’s configuration requires updates to every single config file in every single repository that needs the change made, leading to redundancies, inconsistencies, and maintenance headaches.
As teams grow and workflows scale, these limitations start to hold them back. To build modern, decoupled applications, engineering teams need flexible CI/CD that can ingest changes from anywhere and work across repos, all from a centrally managed set of pipeline configurations.
Introducing multi-repo projects
To better support customers building advanced software at scale, CircleCI has redefined how pipelines work on our platform. Developers and platform teams can now build robust, multi-repo validation systems with:
- Shareable configurations: Your CI/CD configuration files no longer have to live in the same repo as the code they run on. Store them in any repository to enable reusability and simplify maintenance.
- Flexible checkout sources: Define exactly what code your pipeline will operate on, regardless of where it’s stored, for more control and clarity over your builds.
- Cross-repository triggering: Run builds in one repository when changes happen in a different repository.
This modular approach to CI/CD architecture gives organizations the ultimate in flexibility, scalability, and maintainability for their CI/CD pipelines. Let’s consider three common use cases supported by this new approach.
Centralized pipeline management
Developers are an organization’s most expensive asset, and let’s face it: most are not CI/CD experts. Managing, maintaining, and debugging pipeline code can be a time-consuming (read: expensive) and error-prone task. Yet in the traditional model, developers are often left to manage their own pipelines in the repositories they control.
Even in organizations that have adopted platform engineering to abstract away some of the complexities of infrastructure setup and maintenance, making a change to how pipelines run often involves the tedious task of updating configuration files in every repository, one by one. The more complex your project gets, the more difficult it becomes to ensure consistency across teams and repositories.
Multi-repo projects solve these challenges by enabling teams to create a centrally managed hub for CI/CD configurations, decoupled from individual project repositories. This approach allows developers to focus on what they do best—writing code—while platform teams enforce standards, ensure consistency, and maintain scalable workflows. As a result, organizations get:
- More productive developers by freeing them from managing and debugging pipeline code and allowing them to focus on delivering features
- Safer workflows by baking security and compliance checks directly into CI/CD configurations that are leveraged across development teams, reducing the risk of vulnerabilities in the SDLC.
- Increased cost controls by reducing duplication of effort across repositories, minimizing maintenance overhead, and enabling platform teams to optimize resources through centralized governance.
Managing pipelines from a central configuration hub dramatically simplifies the process of setting up and managing pipelines. This not only delivers significant efficiency gains throughout your delivery cycle but also strengthens centralized testing and governance, ensuring consistent enforcement of security, compliance, and quality standards across all projects.
One-to-many test suites
Test automation is essential for maintaining quality and velocity. Yet many teams face roadblocks when trying to implement workflows that span multiple codebases.
Consider a scenario where you have separate codebases for web, Android, and iOS versions of an application. Whenever a new release is tagged in one of these repositories, you want to trigger a shared set of tests across all three platforms.
In traditional CI systems, testing configurations are tightly coupled to the repository where code changes occur. If you want to ensure consistent testing across multiple codebases, you often ry, which leads to maintenance headaches, inconsistencies, and inefficiencies.
By optionally decoupling configurations from individual repositories, CircleCI’s framework allows all changes—whether in the web, Android, or iOS repo—to trigger the same centralized testing suite. This ensures that shared dependencies, APIs, and integrations are validated consistently across all platforms. It eliminates the need for redundant configurations, simplifies testing workflows, and allows for a unified approach to validating the entire ecosystem.
Governance gates
In modern software organizations, delivering code to production is as much about security, compliance, and governance standards as it is about speed. This is especially true in industries such as finance, healthcare, and government, where failure to adhere to these standards can lead to severe consequences.
Many centralized software delivery teams implement governance gates — mandatory checks for security, compliance, and policy adherence — that every application must pass through before change management can even be considered. No app gets a green light for critical environments without passing these checks.
With repo-centric CI/CD, teams are forced to duplicate governance workflows across multiple projects, which can lead to inconsistency and maintenance overhead. CircleCI’s new project framework makes governance gates easier to implement and manage across your organization. Platform and Security teams can now create and own a governance pipeline that all application teams must run before their application gets to production.
Implementing centralized config management on CircleCI
Let’s walk through the process of setting up a centralized repository for managing CI/CD configuration files. In just a few clicks, you’ll be able to create reusable, scalable pipelines that enhance collaboration and simplify governance across your entire organization.
To get started, you’ll need the following resources:
- A CircleCI account
- A GitHub project that has been set up in CircleCI
- One or more projects where your application code lives
Step 1: Set up your config repo
The first step in decoupling pipeline configurations from your project code is to set up a central repository dedicated to storing and managing config files. In our case, the platform team has set up the repository as shown below.
It’s a simple structure: a single folder named templates
that holds the defined configurations for the organization. Right now, we have templates for a Golang project, a Node.js frontend builder, and a Python backend builder. This setup makes it easy to maintain and scale as new configurations are needed.
Step 2: Connect your application to a pipeline
Let’s step into the shoes of a backend team looking to use one of the centrally managed configurations for a Python backend app. Once your CircleCI project is set up for your application code, the next thing you’ll need to do is head over to the project settings. This is where you’ll link your application to pull its CI/CD configuration directly from the centralized config repo, ensuring you’re aligned with the organization’s standards.
From here we will go to the Pipelines tab on the left hand side bar and click Add Pipeline. Pipelines coordinate the execution of workflows and jobs to build, test, and deploy your code.
Step 3: Configure your pipeline
Next, we’ll configure the pipeline to tie everything together. Start by ensuring the CircleCI GitHub App is installed in your GitHub organization, if it hasn’t been already.
Give your pipeline a clear, descriptive name — for instance, Run tests and deploy
.
The next step involves pulling the configuration from the config repo. Here, you’ll need to provide the config source
(name of your centralized config repository) and the exact path to the config file. This is a crucial step, as it relies on the platform engineering team having a well-documented internal process in place. Developers need to understand how to locate and reference the right configuration, especially if they don’t have direct access to view the config repository in the version control system (VCS).
When the pipeline is triggered, CircleCI will fetch the config from the specified path and execute it.
Finally, in the Build Asset section, define the repository for the application code where you want the checkout step to run. This is your Checkout Source
and represents the repo containing the code being built and deployed.
Step 4: Configure pipeline triggers
After saving the pipeline, it’s time to configure our triggers for the project. Triggers are rules to define which events should kick off a pipeline run. A project can have one or more triggers, which are associated with events from a source of change. The most common type of trigger is a VCS event, such as a push to a branch on a GitHub repository. But any system that can emit outbound webhooks can also be set up as a trigger. A trigger can have only one pipeline, but a pipeline can have multiple triggers.
To set up a trigger for your pipeline, select Add Trigger and choose the trigger type from the dropdown. In our case the trigger type is our Github App.
Next, pick a meaningful trigger name that clearly describes the event that causes the trigger. In our case, we named our trigger push to python app
to denote that a change to our Python backend project will trigger our pipeline.
Finally, select the repository with the application code the development team will be iterating on. From there, we can point to the pipeline that we just created. We will then be prompted to choose the branch of the Config Hub that the config should be fetched from whenever you trigger a pipeline.
Congratulations! You’ve successfully configured your application to pull its CI/CD configuration from the centralized config repository.
Extra: Installing the GitHub App on an Oauth account
Initially, this capability was exclusive to projects connected via a standalone organization, but it is now available for Oauth organizations too. If you have been using CircleCI for some time, chances are you are leveraging an Oauth org. This update expands the flexibility of your CI/CD setup, enabling teams to streamline processes across repositories, regardless of how they are connected to CircleCI.
Let’s walk through setting up a centralized configuration repository for your Oauth org. We will use the nodejs-config.yml
that lives in the same centralized repository used above.
⚠️ Prefer a visual walkthrough? You can follow along with the instructions in this video tutorial.
The first step will be to go to project settings and configure a pipeline. A pipeline already exists from when you set up the project using the setup wizard on the projects page. You can ignore this for now and click Add Pipeline.
Choose the GitHub App option from the dropdown menu and click Connect. Then select the organization where you’d like to install the GitHub App.
We will then connect, configure, and authorize the GitHub App. We recommend that you enable access to all repositories.
This saves time and ensures new projects are automatically available on CircleCI without extra steps. However, you can fine-tune the app’s permissions to grant access only to specific repositories.
Once the connection is made we can pass our configuration for the pipeline. I am naming mine build-test-deploy
, pointing the config source to the centralized-circleci-configs
repo and any build assets corresponding to my application code. In this case that is node-frontend-app
.
Now that the pipeline is configured, configure the trigger. Again you’ll notice an Oauth trigger already exists. Ignore that and click Add Trigger.
The trigger type will be GitHub App
.
We can then name the trigger, choose the repository the trigger will act on, the pipeline that the trigger will act on, and finally the branch to pull the configuration from.
The final step is to remove the webhook that had been set up on your repository. To do this you will need to navigate to your GitHub repo, click Settings > Webhooks, and remove the webhook manually. This will prevent a duplication in any pipelines run and ensure that only the config located in the centralized repository will be executed against the trigger configured on the application repo.
⚠️ If you do not complete this step, two triggers will be activated for your pipeline and you will run into an error like the one below.
However, once the webhook has been deleted, only one trigger will be activated, allowing the system to run the pipeline that you configured in this tutorial.
Transform your CI/CD with cross-repo pipelines
By no longer forcing a 1:1 relationship between repos and projects, we’re making it possible to build much more robust, specific CI/CD workflows. Leveraging this functionality, teams can unlock greater flexibility, scalability, and efficiency in their CI/CD workflows.
In this post, we explored three key use cases (centralized config management, centralized testing, and governance gates), showing just a few examples of how platform teams can deliver reusable and scalable pipelines by taking advantage of CircleCI’s new approach to project architecture. We are sure this framework will empower your teams to move faster and build with confidence.
You can learn more about these changes and leave feedback for us in our community forum.