Introducing Test Insights with flaky test detection

The CircleCI Insights dashboard was designed to help you improve your delivery efficiency. We launched the dashboard a year ago to provide teams with actionable data for optimizing your pipelines.

Since then, we’ve been listening to your feedback. By far, the most requested functionality is the ability to gain further visibility into test performance. That’s why we’re officially releasing Test Insights to all CircleCI cloud customers, with plans to extend to server customers in the coming months.

Test Insights provide visibility into the most failed, slowest, and flakiest tests in a given test suite, as well as:

• An expanded lookback window of your recent workflows
• A high-level summary of test suite performance that provides key learnings at a glance
• The ability to test performance across your development branches

Your team can analyze test performance within your most recent executions and identify opportunities for optimization. Access the feature through the Insights dashboard and directly through the tests view for in-flight or completed pipelines.

Flaky test detection

Test Insights includes a new layer of detection and analysis, specifically for flaky tests, which are tests that fail non-deterministically. These types of tests are labeled “FLAKY,” making them easy to scan for on the Insights dashboard.

What’s next for Insights

We believe that observability of test performance is invaluable for all developers, which is why we’re making heavy investments in this area.

We’re currently exploring these improvements to Insights:

• Extending data lookback to support as many recent pipeline executions as possible
• Test Insights for server customers
• Importing all metrics and data through the Insights API
• Flakiness detection improvements that will potentially include manual labeling and custom model support
• Test file aggregations and optimization suggestions

We continue to rely on your feedback to improve. Please submit your ideas via Canny to let us know how you’re using Insights and the new functions you’d like to see.