Static Site Generators (SSGs) such as Hugo and Jekyll are all the rage nowadays. These static on the backend and JavaScript on the frontend websites are called JAMStacks. Typically, we test them in two ways:

  • via the SSG successfully building the site
  • and HTMLProofer

What if we wanted to do more? Let’s walk through a new tool I made for testing markdown files and how it improved the accuracy of CircleCI docs examples.

As a Developer Advocate with CircleCI, a member of the Hugo community, and a frequent contributor to CircleCI docs (built with Jekyll), I’m constantly thinking of ways to improve the testing of static sites. Having a custom and slim Docker image can improve build times and parallelizing jobs when possible helps as well. More raw testing to make sure pages look right and work right is most important but sometimes hardest to do. Frequent feedback from CircleCI customers regarding CircleCI Docs gave me an idea.

The problem

Like most technical docs, CircleCI docs are written in Markdown and include hundreds of examples. These examples are written in GitHub Flavored Markdown fenced code blocks. The majority of our examples are snippets or full CircleCI Configuration files which use the YAML syntax. Many users, when reading through our docs, will see some example YAML that is useful to them and will naturally copy & paste it to their own configs. We then broke their config.

How?

YAML is very strict. Indention means everything. The amount of indention tells the YAML interpretor how to unmarshal all of the data. The wrong indention can make or break a config file. Furthermore, the YAML spec long ago made the–in this writer’s opinion, bad–decision to indent via spaces only. A tab character at the beginning of a YAML line is a syntax error.

Bringing this back to docs, when the CircleCI docs team, other employees, or even community members, contribute to the repo to add or update an example, sometimes these errors creep in. This can be a simple human error–we all make mistakes–or sometimes can be caused by the text editor assuming things when it shouldn’t. Either way, syntax errors which may not be visible to the human eye (tabs characters look like spaces) get merged in and later on a user comes along, copies it, and then has errors. Understandably, this can lead to frustration and there’s no way for HTMLProofer, a great tool, to catch this.

The solution

There are tools other than HTMLProofer out there to test sites and some of them can cover markdown files. To help improve the quality of CircleCI docs, my personal projects, and other static websites out there I’ve written a new tool called Markdown Proofer. It’s a small, open source, Command-Line Interface (CLI) written in Go designed to test markdown files. With CircleCI docs as inspiration, Markdown Proofer’s first and only feature so far is to look for YAML fenced code blocks in markdown files and then validate the YAML to make sure it’s syntactically correct. This allows someone to automate the testing of YAML code blocks in their markdown hopefully leading to fewer error-ridden examples in their site. Let’s test this new tool with CircleCI docs.

Testing CircleCI docs with Markdown Proofer

After pushing Markdown Proofer’s first release, v0.1.0, I created a PR in CircleCI docs to add it to the build process and see what we would catch. As expected, the PR build failed due to several errors that Markdown Proofer caught. You can see this PR on GitHub here and the failing CircleCI build here.

So, great! The tool did its job and found some errors. Now I needed to fix them so that the build would pass and we could officially get Markdown Proofer into CircleCI docs’ master branch. I worked on a new PR to fix all of the YAML errors that were found, and, not counting any superficial changes, was able to fix 11 errors in our YAML examples. This PR can be found on GitHub here.

With all PRs merged, 11 YAML errors were fixed in CircleCI docs’ config examples, future YAML errors will be caught without without having to manually check everything, and only 1 second or less was added to the overall build time. While not groundbreaking, I was able to add a small tool to the CircleCI docs build process in order to improve the quality of docs that we provide to users. Sounds like a win to me.

What’s next?

For Markdown Proofer

Markdown Proofer is still a brand new tool that only checks YAML fenced code blocks. In the near and distant future, I plan to:

  • add support for JSON code blocks.
  • add support for JavaScript code blocks.
  • validate CircleCI Config examples (a complete file) for not only YAML syntax but CircleCI config scheme as well using the CircleCI Local CLI.
  • add more polish to CLI output and README.md.
  • improve error messages (the Go YAML library doesn’t provide great messaging).

You can follow along with the project and open Issues or submit PRs right on GitHub as Markdown Proofer is open source. Find it here.

For static website testing

Many websites are just using HTMLProofer for testing and we can do more. Maybe Markdown Proofer is your thing, maybe not. Testing your site’s UI with Selenium is another way to test most websites. Learn about what’s needed to get better logs of the test process to aid in troubleshooting hung and failed builds. There’s also unique tests for the type of content you have. For example, if you have a general RSS feed, a sitemap.xml file, a podcast feed, etc., there are clever ways to test these pages to make sure they are not just returning an HTTP 200 “Status OK” response, but actually work and make sense.

If you have any ideas of your own or any questions, please continue the discussion on CircleCI Discuss.