Change Failure Rate

⭐️ This metric is one of the 4 Key Metrics published by Google's DevOps Research and Assessment (DORA) team.

You can see all 4 key DORA metrics on the DORA Metrics page of the Multitudes app.

Note: this metric is only shown at a team level, not an individual level.

Change Failure Rate graph

What it is: The percentage of PRs merged that indicate that some kind of failure was released and had to be fixed.

Why it matters: This is our take on DORA's Change Failure Rate, which indicates the percentage of deployments that cause a failure in production. It's a lagging indicator of the quality of software that is being deployed - how often does it contain bugs that cause failures later?

How we calculate it: We calculate the % of merged PRs that contain any of the words rollback, hotfix, revert, or [cfr] in the PR title, out of all merged PRs. We tested and chose these keywords to catch most cases where a PR was required to fix a previously-released bug, while minimizing false positives.

We recognise that this proxies failures after the fact; this is because it's not actually possible to know if someone's releasing a failure into production in the moment, otherwise it wouldn't have been released in the first place!

Also, incidents are not always easily tied to a specific PR or deployment. You can include the square-bracketed keyword [cfr] in your PR titles if you'd like more granular control over what gets counted in this chart.

What good looks like

Google suggests that elite teams should aim for a change failure rate of 0%-5% (originally 0%-15%).