Documentation structure for LLMs (llms.txt)
CircleCI Logo

Persisting data overview

19 days ago
Cloud Server

This guide gives an introductory overview of the various ways to persist and optimize data within and beyond your CircleCI builds. Use artifacts, workspaces, and caches to move data into, out of, and between jobs, persisting data for future use. Using the right feature for the right task will help speed up your builds, and improve repeatability and efficiency.

Note the following distinctions between artifacts, workspaces, and caches:

Type Use Example

Artifacts

Preserve long-term artifacts.

Available in the Artifacts tab of the Job page under the tmp/circle-artifacts.<hash>/container or similar directory.

Workspaces

Attach the workspace in a downstream container with the attach_workspace: step.

The attach_workspace copies and re-creates the entire workspace content when it runs.

Caches

Store non-vital data that may help the job run faster, for example npm or Gem packages.

The save_cache job step with a path to a list of directories to add and a key to uniquely identify the cache (for example, the branch, build number, or revision). Restore the cache with restore_cache and the appropriate key.

Caching

Caching persists data between the same job in different workflows, allowing you to reuse the data from expensive fetch operations from previous jobs. After an initial job run, future pipelines will run faster as they will not need to redo the work (provided your cache has not been invalidated).

Caching dependencies

A prime example of a caching strategy is using a cache with dependency managers, such as Yarn, Bundler, or Pip. When dependencies are restored from a cache, commands like yarn install will only need to download new dependencies (if any), instead of re-downloading every dependency on every build.

Because caches are global within a project, a cache saved on one branch will be used by jobs run on other branches. Caches are only suitable for data that can be shared across branches.

  • For more information on caching dependencies, including race conditions, managing caches, and using cache keys, see the Caching Dependencies page.

Caching optimization

Optimize your pipelines to ensure you are getting the most out of your network and storage usage. For example, when looking for opportunities to reduce data usage, consider whether specific usage is providing enough value to be kept. In the case of caches, consider whether the developer or compute time-saving from the cache outweighs the cost of the download and upload.

Caching optimization strategies can include avoiding unnecessary workflow reruns, combining jobs, creating meaningful workflow orders, and pruning.

  • For more information on caching optimization and other caching strategies, like partial dependency caching, caching tradeoffs, and using multiple caches, see the Caching page.

Workspaces

Workspaces are used to transfer data to downstream jobs as the workflow progresses. When a workspace is declared in a job, files and directories can be added to it. Each addition creates a new layer in the workspace filesystem. Downstream jobs can then use this workspace for their own needs, or add more layers on top.

Workspace optimization

If you notice your workspace usage is high, search for the persist_to_workspace command in your .circleci/config.yml file. This helps identify all jobs using workspaces and determine if all items in the path are necessary.

You also might find that you are only using workspaces to be able to re-run builds from failure. Once the failing build passes, the workspace might not be needed. Setting a low storage period of, for example, one day, might be suitable for your projects. A low storage retention period for workspaces will save costs by not keeping unnecessary data in storage.

Artifacts

Artifacts are used for longer-term storage of the outputs of your pipelines. For example, if you have a Java project, your build will most likely produce a .jar file of your code. This code will be validated by your tests. If the whole build/test process passes, then the output of the process (the .jar) can be stored as an artifact. The .jar file is available to download from our artifacts system long after the workflow that created it has finished.

If your project needs to be packaged, say an Android app where the .apk file is uploaded to Google Play, you would likely wish to store it as an artifact. Many users take their artifacts and upload them to a company-wide storage location such as Amazon S3 or Artifactory.

Artifact optimization

Artifacts can be useful for troubleshooting why a pipeline is failing. However, once the issue is resolved and the pipeline runs successfully, the artifact might serve little purpose. Setting a storage period of, for example, one day, allows you to both troubleshoot the build as well as save costs by not keeping unnecessary data in storage.

If you need to store artifacts for longer periods of time, there are other optimization options available, depending on what you are trying to accomplish. Every project is different, but you can try the following actions to reduce network and storage usage:

  • Check if store_artifacts is uploading unnecessary files.

  • Check for identical artifacts if you are using parallelism.

  • Compress text artifacts at minimal cost.

  • Filter out and upload only failing UI tests with images/videos.

  • Filter out and upload only failures or successes.

  • Upload artifacts to a single branch.

  • Upload large artifacts to your own bucket at no cost.

For more information on artifact optimization, and using artifacts to persist data once a job has completed, see the Storing Build Artifacts page.

Job level retention

Job level retention requires CircleCI Server 4.9 or later.

You can configure job retention periods to control how long job data is kept. Job retention specifically controls cache retention at the job level. This setting can be configured from 1 day to 15 days using string values (for example, "1d", "7d", "15d") in your CircleCI configuration YAML file. This reduces retention from the organization-level default, automatically removing cache data after the specified period.

The following example shows how to configure job retention in your configuration file:

version: 2.1

jobs:
  test:
    docker:
      - image: cimg/node:18.0
    retention:
      caches: 7d
    steps:
      - checkout
      - run: npm install
      - run: npm test

See also