Change Failure Rate

⭐️ This metric is one of the 4 Key Metrics published by Google's DevOps Research and Assessment (DORA) team.

You can see all 4 key DORA metrics on the DORA Metrics page of the Multitudes app.

Note: this metric is only shown at a team level, not an individual level.

Change Failure Rate graph

What it is: The percentage of PRs merged that indicate that some kind of failure was released and had to be fixed.

Why it matters: This is our take on DORA's Change Failure Rate, which indicates the percentage of deployments that cause a failure in production. It's a lagging indicator of the quality of software that is being deployed - how often does it contain bugs that cause failures later?

How we calculate it: We calculate the % of merged PRs that contain any of the words rollback, hotfix, revert, or [cfr] in the PR title, out of all merged PRs. We tested and chose these keywords to catch most cases where a PR was required to fix a previously-released bug, while minimizing false positives.

Tip: Use a script to automatically add [cfr] to PRs that meet certain criteria

In the future we will allow customization of which key words we take into account when determining what constitutes a change failure. In the interim if your team has established practises around naming conventions, you can use a CI script to add [cfr] to the title of PRs you wish to track in Multitudes. This example Github Actions job checks to see if the PR title contains the word fix, and if so appends [cfr] to the title.

name: Multitudes - Change Failure Rate

on:
    pull_request:
        types: [opened, edited] # runs when PR is created or the title is changed

permissions:
    pull-requests: write
    issues: write

jobs:
    update-title:
        runs-on: ubuntu-latest
        steps:
            - name: Adjust title if necessary
              uses: actions/github-script@v7
              with:
                  script: |
                      const pr = context.payload.pull_request;
                      const originalTitle = pr.title;

                      // Rules
                      const hasFixPrefix = /^fix\(/.test(originalTitle);
                      const hasCfrSuffix = /\s*\[cfr\]$/.test(originalTitle);

                      let newTitle = null;
                      let comment = null;

                      if (hasFixPrefix && !hasCfrSuffix) {
                        newTitle = `${originalTitle} [cfr]`;
                        comment = "🤖 **Automated update**: Added `[cfr]` tag to PR title since this is a fix commit. This will help track 'change failure rate' in Multitudes.";
                      } else if (!hasFixPrefix && hasCfrSuffix) {
                        newTitle = originalTitle.replace(/\s*\[cfr\]$/, '');
                        comment = "🤖 **Automated update**: Removed `[cfr]` tag from PR title since this is not a fix commit. The `[cfr]` tag should only be used for fix commits.";
                      }

                      if (newTitle === null) {
                        core.info("No update needed.");
                      }

                      if (newTitle !== null) {
                        await github.rest.pulls.update({
                          owner: context.repo.owner,
                          repo: context.repo.repo,
                          pull_number: pr.number,
                          title: newTitle
                        });

                        await github.rest.issues.createComment({
                          owner: context.repo.owner,
                          repo: context.repo.repo,
                          issue_number: pr.number,
                          body: comment
                        });

                        core.info(`Updated PR title: "${originalTitle}" -> "${newTitle}"`);
                      }

We recognise that this proxies failures after the fact; this is because it's not actually possible to know if someone's releasing a failure into production in the moment, otherwise it wouldn't have been released in the first place!

Also, incidents are not always easily tied to a specific PR or deployment. You can include the square-bracketed keyword [cfr] in your PR titles if you'd like more granular control over what gets counted in this chart.

What good looks like

In Google's DORA research, the top 15th percentile have a Change Failure Rate of 0-4% (see DORA 2025). The DORA benchmark has changed over the years – it was 0%-5% until recently, and 0%-15% before that.