Deployment Metrics

Deployment metrics (e.g., Change Lead Time if you’d like it to include deploy time, Deploy Time, Deployment Frequency, Deployment Failure Rate) require you to either install our GitHub Actions integration or use our Deployments API for other CI/CD tooling.

If you’re using our Deployments API

An “attempt to deploy to production” refers to the POST that you’ve sent to our API. For timestamp, we’ll either use:

• The optional deployedAt value if provided, OR

• The timestamp when we receive your POST call

When naming the deployment, we’ll either use:

• The optional title value if provided, OR

• The commit message of the matched commit with the most recent commit timestamp

Note that we include everything.

For example, if you send us 3 deployments with the same commitSha (let's say because you'd deployed it to dev, staging, and prod), we will show all three in the Deployment Frequency chart. That said, if we receive multiple deployments with the same commitSha, metrics relating to time (e.g., Deploy Time) will only look at the latest attempt.

At the moment, there is not a way to delete specific deployments that you've sent to us. Please email [e-mail] if a POST was made in error.

If you’re using our GitHub Actions integration

The definition of an “attempt to deploy to production” will depend on how you’ve set up GitHub Actions, so be sure to configure your GitHub Actions integration.

1. If you selected "Workflows", this will be an attempt of any of the workflow runs that you've selected in your configuration

• If you have multiple workflows that deploy to prod, all deploys will be considered in your Multitudes metrics. Here are some examples:

  • If you have two workflows, and one fails and one succeeds:

    1. The failed workflow gets counted as part of Deployment Failure Rate

    2. The Successful workflow gets counted as part of Deployment Frequency

    3. Change Lead Time will use the successful workflow run

    4. Deploy Time will use the successful workflow run

  • If you have two workflows, and both succeed:

    1. Change Lead Time will use the earliest successful workflow

    2. Both will be counted as part of Deployment Frequency

    3. Both will be measured in Deploy Time

1. If you selected "Environment/Deployments", this further depends on the metric

• For Change Lead Time (when including deploy time), Deploy Time, and Deployment Failure Rate, this will be an attempt of the workflow run that contains the deployment to the environment/s that you've selected in your configuration

  • If a deployment job successfully deploys code to your selected environment, but something else causes the overall workflow run to fail, then this will not count as a successful deploy

  • In this case, you’ll see a longer Deploy Time and Change Lead Time, since it’s waiting for another successful attempt of both deployment job and overall workflow run

• For Deployment Frequency, this will be an attempt of just the deployment job itself to the environment that you've selected in your configuration

  • If a deployment job successfully deploys code to your selected environment, but something else causes the overall workflow run to fail, then this will still count as a successful deploy

• Here's a concrete example of these nuances:

  • You have a workflow called deploy-code, which includes these jobs

    1. build-code

    2. apply-terraform, which works on the prod Environment you selected in your configuration step, and therefore represents the point at which your change is now shipped to users)

    3. send-update-to-slack, an extra internal notification step

  • To deploy to production, you run deploy-code, and the jobs build-code and apply-terraform succeed. Your code is now out to users in the production environment.

  • However, send-update-to-slack fails

  • This will count toward Deployment Frequency, because your code actually did get deployed to production

  • This will not count toward Deploy Time, or Change Lead Time through to deployment, and will register as a failure in Deployment Failure Rate because the job that failed still disrupts the smooth flow of CI/CD (i.e., developers have to investigate)

Lastly, to align on the definitions of success and failure:

• Successful attempt = attempt concluded with SUCCESS

• Failed attempt = attempt concluded with FAILURE , TIMED_OUT , or STARTUP_FAILURE

• We ignore other conclusions (i.e., CANCELLED or SKIPPED)

Note that these definitions apply regardless of the nuance expressed earlier across what an attempt is based on how GitHub Actions is used and configured.