Change Failure Rate

⭐️ This metric is one of the 4 Key Metrics published by Google's DevOps Research and Assessment (DORA) team.

You can see all 4 key DORA metrics on the DORA Metrics page of the Multitudes app.

Note: this metric is only shown at a team level, not an individual level.

Change Failure Rate graph

What it is: The percentage of PRs merged that indicate that some kind of failure was released and had to be fixed.

Why it matters: This is our take on DORA's Change Failure Rate, which indicates the percentage of deployments that cause a failure in production. It's a lagging indicator of the quality of software that is being deployed - how often does it contain bugs that cause failures later?

How we calculate it: We calculate the % of merged PRs with specific keywords in the title.

Specifically, we look at:

(1) the number of merged PRs that contain any of the words rollback, hotfix, revert, or [cfr] in the PR title

and then divide by

(2) the total count of merged PRs (including selfie PRs, which are PRs merged by author with no comments or reviews from others).

We tested and chose the keywords above to catch most cases where a PR was required to fix a previously-released bug, while minimizing false positives. When there was disagreement about whether to include a keyword (e.g., fix), we excluded it – so for most teams, our Change Failure Rate is a low estimate.

Tip: Use a script to automatically add [cfr] to PRs that meet certain criteria

In the future we will allow customization of which key words we take into account when determining what constitutes a change failure. In the interim if your team has established practises around naming conventions, you can use a CI script to add [cfr] to the title of PRs you wish to track in Multitudes.

Example 1: Github Actions job that checks if the PR title contains the word fix, and if so appends [cfr] to the title.

name: Multitudes - Tag Fix PRs for Change Failure Rate 

on:
    pull_request:
        types: [opened, edited] # runs when PR is created or the title is changed

permissions:
    pull-requests: write
    issues: write

jobs:
    update-title:
        runs-on: ubuntu-latest
        steps:
            - name: Adjust title if necessary
              uses: actions/github-script@v7
              with:
                  script: |
                      const pr = context.payload.pull_request;
                      const originalTitle = pr.title;

                      // Rules
                      const hasFixPrefix = /^fix\(/.test(originalTitle);
                      const hasCfrSuffix = /\s*\[cfr\]$/.test(originalTitle);

                      let newTitle = null;
                      let comment = null;

                      if (hasFixPrefix && !hasCfrSuffix) {
                        newTitle = `${originalTitle} [cfr]`;
                        comment = "🤖 **Automated update**: Added `[cfr]` tag to PR title since this is a fix commit. This will help track 'change failure rate' in Multitudes.";
                      } else if (!hasFixPrefix && hasCfrSuffix) {
                        newTitle = originalTitle.replace(/\s*\[cfr\]$/, '');
                        comment = "🤖 **Automated update**: Removed `[cfr]` tag from PR title since this is not a fix commit. The `[cfr]` tag should only be used for fix commits.";
                      }

                      if (newTitle === null) {
                        core.info("No update needed.");
                      }

                      if (newTitle !== null) {
                        await github.rest.pulls.update({
                          owner: context.repo.owner,
                          repo: context.repo.repo,
                          pull_number: pr.number,
                          title: newTitle
                        });

                        await github.rest.issues.createComment({
                          owner: context.repo.owner,
                          repo: context.repo.repo,
                          issue_number: pr.number,
                          body: comment
                        });

                        core.info(`Updated PR title: "${originalTitle}" -> "${newTitle}"`);
                      }

Example 2: Github Actions job that checks if the PR branch contains the word hotfix, and if so appends [cfr] to the title.

name: Multitudes - Tag Hotfix PRs for Change Failure Rate

on:
    pull_request:
        types: [opened, edited]

permissions:
    pull-requests: write
    issues: write

jobs:
    update-title:
        runs-on: ubuntu-latest
        steps:
            - name: Adjust title if necessary
              uses: actions/github-script@v7
              env:
                  BRANCH_PREFIX: "hotfix"
              with:
                  script: |
                      const pr = context.payload.pull_request;
                      const headBranch = pr.head.ref;
                      const originalTitle = pr.title;
                      const branchPrefix = process.env.BRANCH_PREFIX;

                      // Rules
                      const hasMatchingBranch = new RegExp(`^${branchPrefix}/`).test(headBranch);
                      const hasCfrSuffix = /\s*\[cfr\]$/.test(originalTitle);

                      let newTitle = null;
                      let comment = null;

                      if (hasMatchingBranch && !hasCfrSuffix) {
                        newTitle = `${originalTitle} [cfr]`;
                        comment = `🤖 **Automated update**: Added \`[cfr]\` tag to PR title since this is a ${branchPrefix} branch. This will help track 'change failure rate' in Multitudes.`;
                      } 

                      if (newTitle === null) {
                        core.info("No update needed.");
                      }

                      if (newTitle !== null) {
                        await github.rest.pulls.update({
                          owner: context.repo.owner,
                          repo: context.repo.repo,
                          pull_number: pr.number,
                          title: newTitle
                        });

                        await github.rest.issues.createComment({
                          owner: context.repo.owner,
                          repo: context.repo.repo,
                          issue_number: pr.number,
                          body: comment
                        });

                        core.info(`Updated PR title: "${originalTitle}" -> "${newTitle}"`);
                      }

We recognize that this proxies failures after the fact; this is because it's not actually possible to know if someone's releasing a failure into production in the moment, otherwise it wouldn't have been released in the first place.

You can include the square-bracketed keyword [cfr] in your PR titles if you'd like more granular control over what gets counted in this chart.

The term "selfie PR" refers to pull requests that were merged by the author with no comments or reviews from others (excluding comments and reviews from bots)

What good looks like

In Google's DORA research, the top 15th percentile have a Change Failure Rate of 0-4% (see DORA 2025). The DORA benchmark has changed over the years – it was 0%-5% until recently, and 0%-15% before that.