Automate GitHub stats reporting with scheduled pipelines

This tutorial covers:

1. Setting up a Node.js app
2. Generating GitHub reports
3. Writing, running, and scheduling a CircleCI pipeline

Release notes provide essential documentation when a new software version is released. For release notes to be most effective, dev teams must consolidate all of the work that has been done since the previous release. It is a hectic task that requires a lot of effort and time sorting through weeks or even months of software issues and pull requests.

Why not make the life of the release team easier by automating the creation of release notes? You can, using a combination of GitHub API and a CI/CD tool like CircleCI. Automate the task of fetching issues and pull requests and put them in a single place where they can be accessed easily by the release notes team.

In this tutorial, you’ll learn to use the GitHub API and CircleCI to create weekly stats for your GitHub repositories. The plan is to build an automated workflow using CircleCI scheduled pipelines. The pipeline will fetch all the issues and pull requests made during a specified interval, save these stats in a file, and commit this file back to the repository.

Prerequisites

To follow along with this tutorial, you’ll need:

• Node.js v14.19.0
• CircleCI account
• GitHub account

Our tutorials are platform-agnostic, but use CircleCI as an example. If you don’t have a CircleCI account, sign up for a free one here.

Setting up a Node.js app

The first thing you need to set up is a Node.js app. This is where you’ll write the code to create the weekly reports related to issues and pull requests. From your terminal, go to the path you want and run:

mkdir github-stats

For this tutorial, you need to install the following NPM dependencies:

• The @octokit/rest library serves as the GitHub REST API client for JavaScript/TypeScript projects.
• The fs library provides functions to interact with the file system.

To install these dependencies, go to the github-stats directory and run these commands:

cd github-status
npm init
npm install @octokit/rest fs

When the dependencies are installed, open the package.json file and add the type property to load the project as an ES module:

{
  ...,
  "type": "module"
}

Create an src directory in the github-stats directory. In the src directory, create an index.js file. In the index.js file, you’ll write the code for generating reports.

Generating GitHub reports

In this section, you’ll write the code for creating reports by getting the required data from GitHub using the GitHub API.

Open the src/index.js file and add this code to it:

// 1
import { Octokit } from "@octokit/rest";
import fs from "fs";

// 2
const REPO_NAME = "octocat/Hello-World";

// 3
const octokit = new Octokit();

// 4
const d = new Date();
d.setDate(d.getDate() - 7);

// 5
const checkAfter = d.toISOString().split("T")[0];

This code:

1. Imports the required NPM modules.
   • GitHub account
2. Defines the name of the repository (REPO_NAME) you want to create the reports from. For this tutorial, we have used the octocat/Hello-World repository.
3. Initializes the octokit object using the Octokit module to interact with GitHub.
4. Creates a date object (d) and sets its value to seven days (one week) before the current date. You can change this value if you want to. For example, if you want to create monthly reports, you can subtract 30 from the current date.
5. Converts the date (d) to YYYY-MM-DD format, which is required by GitHub API when setting up the date filters in the search query.

Next, add the code for fetching the issues:

...

const fetchIssues = async () => {
  // 1
  const q = `repo:${REPO_NAME} is:issue updated:>${checkAfter}`;

  // 2
  const issuesAndPullRequests = await octokit.rest.search.issuesAndPullRequests(
    { q }
  );

  // 3
  let issuesContent = "";

  // 4
  issuesAndPullRequests.data.items.forEach((item) => {
    issuesContent += `${item.title} - ${item.html_url}\n\n`;
  });

  return issuesContent;
};

This code:

1. Uses the fetchIssues function, to define the search query (q) for fetching the issues (is:issue) in the specified repo (repo:) that were updated after a particular date (updated:>). You can learn more about search query building from the GitHub API Docs.
2. Calls the issuesAndPullRequests method from the octokit library and passes to it the query parameter (q).
3. Initializes the issuesContent variable to store the content related to issues that you’ll use while creating the final report.
4. Iterates over the response received from GitHub API and adds it to the issuesContent string variable. For this tutorial, we have used the title and html_url properties.

Next, add the code for fetching the pull requests:

...

const fetchPRs = async () => {
  // 1
  const q = `repo:${REPO_NAME} is:pull-request updated:>${checkAfter}`;

  // 2
  const issuesAndPullRequests = await octokit.rest.search.issuesAndPullRequests(
    { q }
  );

  // 3
  let prsContent = "";

  // 4
  issuesAndPullRequests.data.items.forEach((item) => {
    prsContent += `${item.title} - ${item.html_url}\n\n`;
  });

  return prsContent;
};

This code:

1. Uses the fetchPRs function to define the search query (q) for fetching the pull requests (is:pull-requese) in the specified repo (repo:) that were updated after a particular date (updated:>).
2. Calls the issuesAndPullRequests method from the octokit library and passes to it the query parameter (q).
3. Initializes the prsContent variable to store the content related to pull requests that you’ll use while creating the final report.
4. Iterates over the response received from GitHub API and adds it to the prsContent string variable. For this tutorial, we have used the title and html_url properties.

Next, add the code for the main function of the Node.js app:

...

const main = async () => {
  // 1
  const reportTemplate = `# Daily Report - {{date}}

## Issues

{{issues}}

## PRs

{{prs}}
`;

  // 2
  const issuesContent = await fetchIssues();
  const prsContent = await fetchPRs();
  const today = new Date().toISOString().split("T")[0];

  // 3
  const reportContent = reportTemplate
    .replace(/{{issues}}/g, issuesContent)
    .replace(/{{prs}}/g, prsContent)
    .replace(/{{date}}/g, today);

  // 4
  const reportsDir = "./reports";

  // 5
  if (!fs.existsSync(reportsDir)) {
    fs.mkdirSync(reportsDir);
  }

  // 6
  fs.writeFileSync(`${reportsDir}/${today}.md`, reportContent);
};

// 7
main();

This code:

1. Uses the main function, to define the template for your report (reportTemplate) and uses the moustache syntax for issues, pull requests, and the report’s date.
2. Gets the values for issues (issuesContent), pull requests (prsContent) and today’s date (today).
3. Creates the report (reportContent) by replacing the required values in the report template (reportTemplate).
4. Defines the path of the directory (reportsDir) where you want to create the reports. For this tutorial, we have used the reports directory in the project’s root directory.
5. Creates the reports directory if it doesn’t exist using mkdirSync method from the fs module.
6. Creates a markdown file and adds the report’s content (reportContent) to it using the writeFileSync method from the fs module.
7. Calls the main function to execute when you run the node src/index.js command.

That’s it. The coding part is complete! The next step is to set up the configuration for CircleCI builds.

Writing a CircleCI pipeline

In this section, you will write a script to run the report generation code in CircleCI.

At your project’s root directory (github-stats), create a .circleci directory and add to it a config.yml file.

Next, open the .circleci/config.yml file and add this configuration to it:

version: 2.1
jobs:
  build:
    working_directory: ~/repo
    docker:
      - image: cimg/node:14.19.0
    steps:
      # 1
      - checkout
      - run:
          name: Update NPM
          command: "sudo npm install -g npm"
      - restore_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}

      # 2
      - run:
          name: Install Dependencies
          command: npm install
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - ./node_modules

      # 3
      - run:
          name: Generate Reports
          command: node src/index.js
      - save_cache:
          key: reports-public-cache-{{ .Branch }}
          paths:
            - ./reports

# 4
workflows:
  version: 2
  build-deploy:
    jobs:
      - build:
          filters:
            branches:
              only:
                - master

This configuration:

1. Checks out the project from the repository.
2. Installs and caches the project dependencies.
3. Runs the Node.js app (node src/index.js) to generate a report.
4. Configures the workflow to run only when a push is made to the master branch.

Finally, push your code to GitHub using these git commands:

git init
git remote add origin <YOUR_GITHUB_REPOSITORY_URL>
git add .
git commit -m "feat: initial commit"
git push origin master

Next, you need to set up CircleCI to run the CI/CD pipelines.

Setting up CircleCI

To integrate CircleCI with your repository, log in.

Next, click Projects in the left sidebar menu on the CircleCI dashboard.

Find your project in the list and click Set Up Project. CircleCI will detect your configuration file. Give it a quick review and click Set Up Project.

This triggers a build on the CircleCI pipeline. Visit the dashboard to confirm that the pipeline was successful.

If the build is successful, expand the Generate Reports tab to make sure that the main code is working.

Now that the build completes successfully, commit the generated report back to your GitHub repo.

Committing reports to GitHub repository

To commit the generated reports back to the repository, CircleCI needs write access to push changes from the CircleCI pipeline to your GitHub repository.

Create an SSH key to allow CircleCI to authenticate to the GitHub repository by running this command:

ssh-keygen -t rsa -b 4096 -C "CircleCI Deploy Key with Write Access" -m PEM

You’ll be asked for the name of the file. You can specify any name you like. For this tutorial, set the name of the file as circleci_rsa.

The command will create two keys/files:

• Public key (circleci_rsa.pub)
• Private key (circleci_rsa)

Go to your repository on GitHub. Click Settings > Deploy keys on the left menu. There will be a read-only key added by CircleCI to GitHub. To create a key with write access, click Add deploy key.

Next, fill out the required details and paste the public key (circleci_rsa.pub) in the Key field. Check Allow write access to provide CircleCI with write permissions.

You’ll see the new key added under the Deploy keys section.

Next, click Project Settings > SSH Keys under your CircleCI project.

Under the SSH Keys section, scroll down to the Additional SSH Keys section and click Add SSH Key.

On the Add an SSH Key form, enter github.com as the Hostname and paste the private key (circleci_rsa) in the Private Key field. Make sure to keep the private key safe and don’t share it with anybody.

After adding the private key to CircleCI, you will get the fingerprint for the SSH key.

Next, add the generated fingerprint to the CircleCI configuration file .circleci/config.yml in your repository using the stepadd_ssh_keys:

---
- checkout
- add_ssh_keys:
    fingerprints:
      - "<YOUR_FINGERPRINT>"
- run:

Note: Don’t forget to replace the <YOUR_FINGERPRINT> with your value.

Following the Generate Reports step, add the Commit to GitHub Repo step to commit and push the reports back to the master branch of the GitHub repository:

- run:
    name: Generate Reports
    command: node src/index.js
- save_cache:
    key: reports-public-cache-{{ .Branch }}
    paths:
      - ./reports
- run:
    name: Commit to GitHub Repo
    command: |
      git config user.name "CircleCI Job"
      git config user.email noreply@github.com
      git checkout master
      if [ $(git status reports --porcelain=v1 2>/dev/null | wc -l) != "0" ]; then
        git add .
        git commit -m "chore: weekly report from CircleCI [skip ci]"
        git push origin master --force
      fi

Note: The [skip ci] label in the commit message is used to prevent re-triggering of the CircleCI pipeline when a commit is made by the pipeline itself.

That’s it. Both the code and CI setup are complete. Next, you need to test the entire workflow.

Running the CircleCI pipeline

Next you will re-run the CircleCI pipeline to make sure that CircleCI is able to commit generated reports back to the GitHub repository.

Re-trigger the CircleCI pipeline either manually or by creating and pushing an empty commit to your GitHub repository.

Once the pipeline completes, visit your GitHub repository and check out the reports directory.

Open the reports directory. There will be a markdown file: 2022-07-23.md.

Open the markdown file to review a report with issues and pull requests.

With that, you have successfully committed the report back to your GitHub repository. The last step in this tutorial is running the pipeline on a recurring basis.

Scheduling the CircleCI pipeline

Now that you have configured the pipeline correctly, you can run the pipeline automatically on a weekly schedule. Use CircleCI’s pipeline triggers to trigger the pipeline’s execution on a recurring basis.

Go to Project Settings > Triggers. Next, fill out the details related to your trigger. You are free to specify values according to your choice but for this tutorial, use the following values:

• Trigger Source: Scheduled
• Repeats on these days: Saturday
• Start Time (UTC): 11:00 UTC
• Repeats Per Hour: 1
• Branch Name: master

Click Save Trigger to add your trigger.

That’s it. Wait for the trigger to run and you will have a new report in your GitHub repository.

Conclusion

In this tutorial, you learned to create weekly reports for GitHub repositories using GitHub API and CircleCI. You learned about automated scheduled pipelines in CircleCI, and how to commit assets back to the GitHub repository from the CircleCI pipelines.

If you would like to expand on what you have read here, try our tutorial about triggering the report generation pipeline using CircleCI webhooks.