Benefits of scheduled pipelines in continuous integration

Scheduling is an integral part of software development practices. Tools for scheduling jobs help development teams save time by scheduling recurring tasks — like modifying a database or sending out periodic emails — for execution at specified times. There are many to choose from, including cron for Linux, scheduled tasks for Windows, launchd for macOS, Jobber, and anacron. In most cases, a user writes a script with setup instructions, and a command executes the script at the designated time.

Continuous integration and continuous delivery (CI/CD) is another important tool that allows developers to automate recurring processes like building, testing, deploying, and releasing software. CI/CD pipelines are often triggered by a specific event like a new commit to the codebase. But there are many situations where teams can benefit from running builds, tests, and even deployments according to a set schedule instead of as a response to a change event.

Scheduled pipelines give you all the functionality of a regular CI/CD pipeline plus the ability to schedule task execution. In this post, I will explain some of the benefits of scheduling jobs in your continuous integration pipelines. I will also describe some common use cases that you can implement to improve your workflow and optimize your team’s development velocity.

Scheduling jobs in a continuous integration pipeline

Automation is the foundation of CI/CD. Many CI/CD pipelines contain all the necessary steps to automatically bring code from commit to release: build, standardize, test and validate, package, release, deploy, and monitor.

The most significant benefits of CI/CD are improved development speed and collaboration. But without automation, your team won’t be able to realize the benefits. Instead, they will be manually standardizing builds to ensure just one “latest” software version to prevent working on multiple versions simultaneously. You must manually maintain current documentation of all artifacts, dependencies, and software performance reports.

CI/CD pipelines consist of a queue of jobs, often known as workflows, that need to be executed to keep your software secure, functional, and up-to-date. Because automation is mission-critical, how and when you trigger pipelines becomes equally crucial. The ability to trigger your pipeline whenever it can deliver the most value to you and your team is paramount for operational success.

For optimum pipeline triggering, the traditional style of chain-reaction execution is not enough. There are many instances where teams need the flexibility to execute jobs according to a set schedule rather than in response to an event trigger. So, scheduled jobs are a natural fit in a CI/CD pipeline.

There are three categories of jobs that most often benefit from a scheduled pipeline:

1. Deploying artifacts
2. Testing
3. Maintenance Let’s further explore these three types of scheduled CI/CD work.

Deploying artifacts

Deploying artifacts is one part of CI/CD work that benefits from automation and scheduling. When you manually deploy artifacts, you are responsible for consistently establishing and maintaining the deployment schedule.

Automatic scheduling enables you to deploy your artifacts consistently and punctually. This is particularly important when you need to deliver a build on time so that users get the software when they expect it.

For example, you might schedule an automatic release of a build each night for internal or external testers — or for your beta testers who want to try the latest version and don’t mind encountering bugs.

For more information on scheduling nightly builds of your applications, check out Getting started with scheduled pipelines.

Testing

Automating the process of testing your new build is at least equal in importance to automating its deployment. Manual testing wastes significant resources because it is seldom efficient enough to thoroughly and consistently validate every build.

You can use scheduled jobs to run any tests, from QA to the entire test suite. You can be confident in your software’s functionality the next day. Scheduling testing jobs lets you consistently run time-consuming tests that would be inefficient to run for each new commit.

By scheduling tests, you can run them at a consistent time that best suits your team’s needs.

Maintenance

Maintenance relies heavily on user-generated data and software performance metrics. To get these, you must carefully monitor your app and run regular scans. Even if your CI/CD workflow does not continuously deploy, the frequency with which updates are released means that you must find a way to automatically monitor your software in production — especially in the early stages of release.

By scheduling maintenance, you can keep the necessary performance and security scans running on a regular, timely basis. This systematic process gives you one holistic workflow and provides full visibility of your project.

Use cases for scheduled pipelines

There are many and varied use cases that benefit from scheduling pipelines, including:

• Security checks outside the normal commit-push cycle
• Refreshing and resetting resources
• Running regular QA tests on builds
• Data cleanup
• Load testing
• Sending reports and notifications

Security checks outside the normal commit-push cycle

With increasingly frequent and severe cyber threats, securing your application from bad actors has become a crucial component of software development. This process starts when the software is architected and continues endlessly.

Most CI pipelines are triggered to run only when the codebase receives new commits. For most organizations, these commits tend to occur several hours apart. With the frequency at which vulnerabilities are unearthed — whether discovered in your software or a component of your software supply chain — this is no longer adequate. You must run frequent, thorough security scans at regular intervals.

Consistent testing often requires running automated security-based tests. In CI/CD, the best practice is to build these security tests into your CI/CD workflow.

However, you may prefer running these security scans separately from other automated testing for performance and function. Also, given the myriad of security tests that must be performed, scans might need to run more frequently than those triggered by new builds. This is especially true in industries with strict software compliance requirements and when managing critical applications, like payment platforms or apps that collect personal information.

In these cases, it is vital to schedule your scans for set times or intervals instead of using the traditional domino-style approach often used for CI/CD pipeline executions. For an example of how to schedule a security scan using scheduled pipelines, check out Running regular security scans with scheduled pipelines.

Refreshing/resetting resources

Resetting and refreshing resources also benefit from automated scheduling. You can select a time to reset services and environments to a default state, archive logs, and redeploy applications as part of your CI/CD pipeline. Since your CI/CD pipelines can already access your environments, there is no real drawback to using this functionality.

Running regular QA tests on builds

Arguably, there is no CI/CD without automated tests. While manual tests are important, automated tests are the only way that your testing team can match the developmental pace of CI/CD.

Often, organizations automate periodic testing using servers dedicated to QA tests for multiple projects. For this approach to work, builds must be deployed to these servers at set times. For manual tests, an authorized team member must be available to trigger the test, before the build can deploy to the test servers.

With a scheduled pipeline, you can absorb this process into your CI/CD pipeline. You can schedule the deployment of builds to your test servers at a designated time, which automatically triggers the test scripts to run without compromising your established access controls, or requiring a team member’s intervention.

Data cleanup

If your workflow involves data processing, you will commonly execute tasks like data sorting, cleaning, transformation, merging, modeling, database backups, and batch processing. Although these tasks may not be builds, they are still crucial to your workflows and they have a place in your pipelines. With non-scheduled pipelines, it can be challenging to add these tasks to workflows, because they often must run periodically, instead of waiting for an event trigger.

Enter scheduled pipelines. When you can set tasks to run at predefined times, your data workflow becomes part of your project pipeline. For an example of how you can use scheduled pipelines to maintain and update your databases, check out Automating database cleanup with scheduled pipelines.

Load testing

Teams depend on load testing to show that their application meets the expected performance standard. Load testing is the most reliable way to understand how a product will behave under certain real-life conditions.

In CI/CD, load testing is part of the validation criteria for every build. It just makes sense to make these tests a part of your pipelines. Unfortunately, manual load testing is difficult to coordinate and measure and expensive to implement. In contrast, automated load testing lets you schedule these tests to run when it best suits your team, using your resources more effectively.

For example, you can schedule load tests for off-peak work periods to save resources. Alternatively, you can schedule your tests to run during peak use times to obtain real-world or field-testing results. You can learn how to schedule recurring API load tests in Scheduling load tests and persisting output with k6.

Sending reports and notifications

Another major benefit of scheduled pipelines is the ability to keep team members and other stakeholders updated on the status of applications and other important resources. By running your pipelines on a schedule, you can not only deploy build artifacts to staging areas for review, but you can regularly send out emails, update shared dashboards, publish documentation, and perform many other types of reporting jobs that can help the rest of your team feel confident in the work that you are doing. For an example of how you can use CI/CD and scheduled pipelines to automatically generate and deliver invoices on a regular cadence, check out Build an automated invoice generator application.

Benefits of scheduled pipelines

As the use cases in the previous section highlight, scheduled pipelines provide an ideal complement to standard event-based CI/CD pipelines. With scheduled pipelines, you can avoid job queues for nonessential or resource-intensive work and schedule them to run at the most suitable time.

Scheduled pipelines also help you to maintain your application’s security throughout the entire development process. Application security is crucial, and detecting vulnerabilities early can reduce your mean time to recovery (MTTR). To detect vulnerabilities that appear between normal and deploy cycles, you must consistently run security scans and pay attention to their results. Scheduled pipelines provide improved protection in addition to resource management.

Finally, scheduled pipelines are useful for keeping managers, team members, and other stakeholders aware of where to find the newest build, performance data, documentation, and other artifacts.

Conclusion

Delivering the best software experience to users fundamentally relies on the efficiency of the CI/CD pipelines that manage your build and production processes. There are instances where you might need to execute certain tasks — like periodic security checks and QA tests — on a schedule instead of when an event occurs. In these cases, your pipelines must do more than just wait for a trigger.

CircleCI’s scheduled pipelines let you design pipelines that can execute tasks from an event trigger or run them at a time that best meets your operational model and your development team’s needs.

When you’re ready to start transforming your CI/CD workflows, you can sign up for a free CircleCI account and learn more about how to start using scheduled pipelines.