# Welcome ## Multitudes is built for engineering teams who want to lead in AI. Caught between the AI hype and reality? Get insights to see what’s working or not – including tracking the impact of the experiments you run. Create an account, integrate your tools, and act on insights in minutes. {% embed url="" %} ## Discover Multitudes
Introduction to MultitudesLearn how we help engineering teams and how we're different from other analytics tools./pages/bUoytPx8AodTnnEnix85/files/ywGewJbN3baL1q5TodLU
How Multitudes worksEverything you need to know about how Multitudes work./pages/3nssGhxMGsxLt59UUmgM/files/XaLBFQuHcDoXxhxJzNjm
Metrics & DefinitionsThe research-backed metrics we use and how we conduct the analysis that underlies our insights./pages/IxaoAQE0qyP0iJtwFouT/files/JvplbcwLMwPs4AbkV8or
Configuration & SetupSet up and configure your teams on Multitudes./pages/BrZxKHztMoXKlMBe6Eno/files/cja6J952s7h8EkCnynJa
IntegrationsConnect and configure Multitudes with your tools./pages/QfImI085wDmfM9oTOVEh/files/EwiRMHrufSIlG3ZpFRlw
Knowledge baseEssential guides and best practices to help you get the most out of Multitudes./pages/LIuwFSOMOEnfMCZNpwg2/files/mEmZUCyXO89b4oLA4jBa
Account managementLearn about our security practices and how to manage your billing./pages/rmkUK7hNecCF7XNWte8z/files/JK1ByesySvo93xBsSqtp
# Introduction to Multitudes Learn how we help engineering teams and how we're different from other analytics tools. ## What is Multitudes? Multitudes is a software tool that provides analytics and recommendations to unlock happier, higher-performing teams. We integrate with the collaboration tools you use at work – e.g., for version control, CI/CD, issue tracking, or incident management – and then pull out research-backed insights about where work is blocked, how your team is collaborating, who’s at risk of burnout, and more. We bring metrics like DORA, SPACE, and DevEx to life with nudges in the daily workflow, so you can help your team take action. ## Who is Multitudes for? Multitudes is for human-centric engineering teams who want research-backed metrics to improve together. It’s for teams that want the holistic view – not just of team performance, but also about people dynamics like collaboration and wellbeing. Our insights and actions are useful to all members of an engineering organization, whether you’re an engineer, manager, or CTO. That said, our #1 focus is on empowering engineering teams to take action – because the best way to get continuous improvement is to build a daily habit across teams. ## Our approach ### We are not another creepy monitoring tool We get it – a lot of us have been burned by tools that reduced people down to lines of code written, PR's opened, or some blackbox “efficiency score”. We don't do that – not only are these measures not useful, they can be harmful for people. Instead, our focus is on helping teams create the right conditions for people to do great work. Are people getting enough support? Is anyone working long hours? When we do look at “performance” measures, it’s at a team level – for example, looking at the Lead Time for the whole team. PRs are a team sport, and we know that individuals do their best work when they’re in an environment that’s supportive and good for people’s wellbeing. **We're careful about how we show individual metrics, and only when it can bring value to the person who the data is about.** You can learn more about [Metrics & Definitions](/metrics-and-definitions/multitudes-insights). We also hold ourselves (as a team) to rigorous standards of ethics and consent. **We design for transparency; managers and team members see all the same data and insights.** We’ve publicly shared our [data ethics principles](https://www.multitudes.co/blog/measure-what-matters-and-dont-be-creepy), and we welcome feedback on how we’re doing. And finally, if you’re someone who’s been burned in the past and don’t have the energy for a tool like this, we get that too! If someone on the team feels uncomfortable with Multitudes, we recommend that the team not use us. Our goal is to support team trust and collaboration, so we recommend that the team make a unified decision on whether our tool is a fit. ### We are different from other engineering effectiveness tools At Multitudes, we know that PR's are a team sport. People dynamics are what determine team performance, so even if we only wanted to improve team performance, we would still need to provide insights & coaching on how the team works together. As evidence of this, see [Google’s Project Aristotle](https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html) research, which showed that psychological safety is the number one determinant of team performance. That’s why we look at things like collaboration trends and wellbeing on top of the team performance metrics that you often see in other tools. Our second key point of difference is our focus on empowering teams, not on micromanaging them. Our goal is to give teams, including individual contributors, the insights they need to make their own decisions about how to work together. As part of that, we encourage transparency – everyone on the team should have access to their own Multitudes data. We do this for our own team, and we've seen how useful it is for building trust and supporting behavior change. ### We don't ingest your codebase Multitudes pulls in code metadata from GitHub; our app does not ingest your codebase. This metadata includes information about pull requests – such as when they were created, who the author was, commits made on the pull request, number of lines changed, etc. and about comments – including comments written on pull requests and reviews submitted. When you set up the GitHub installation, you can see the full list of permissions that Multitudes requests. Multitudes also gets some data from teams about their structure, e.g., which teams people are on, their roles, and their levels. # How Multitudes Works Learn how Multitudes collects and analyzes your team's data, and how you can use our features to improve your engineering team's performance. Multitudes integrates with the collaboration tools you already use, like GitHub for git management, Github Actions for CI/CD, or Linear and Jira for issue tracking. From there, we pull in metadata on Pull Requests (PRs), as well as data from comments, reviews, and tickets. We do not ingest your actual codebase. Our metrics are based on research including [DORA](https://cloud.google.com/blog/products/devops-sre/dora-2022-accelerate-state-of-devops-report-now-out) and [SPACE](https://queue.acm.org/detail.cfm?id=3454124), and detect outliers and trends that matter for your team – across team performance, collaboration, and wellbeing. To give you a quick overview, we highlight key insights at-a-glance on the homepage and pair them with actions you might want to take. ## Multitudes helps you take action Multitudes automatically suggests actions you might want to take with your team, based on trends in your team’s data. The following two features on our app’s homepage can get you taking action as soon as you log in. All team members have access to both features, but we’ve highlighted what might be useful to whom, based on our research. 1. ***At a glance*** view – Most useful for managers of managers, senior leaders, and executives

At a glance

This is the first thing you’ll see at the top of the homepage when you log in. It is a higher-level overview representing how things are going across teams and the organization over the last 6 weeks. Here’s a more [detailed explanation](/metrics-and-definitions/our-approach-to-metrics) of how this insight is calculated and impacted by custom targets. The text at the bottom of this *At a glance* table shows when the thumb icon statuses and *Actions* were last updated. Each team’s row will also have a dynamically generated *Action* in the last column at right. You can click on *Action* to access the associated *Facilitation Guide*.

Facilitation guide

Each *Facilitation Guide* includes data about the metric we’ve highlighted, ideas for questions you could ask to get more context, and potential experiments to help the team take action. These data-driven insights can add value to your team discussions, such as retros, around process improvements. 2. ***Trend Summary*** section – Most useful for teams and team leads

Trend summary

On the *My Insights* page under the *At a glance* section is the *Trend Summary*, a quick overview of our top 5 metrics organized as cards. Each card has a thumb icon status representing how it’s tracked over the selected date range. Here’s a more [detailed explanation](/metrics-and-definitions/our-approach-to-metrics) of how this insight is calculated and impacted by custom targets. Note that this thumb icon status is calculated differently, and may show a different value than the one in the *At a glance* table above. It also has a *Take Action* section below the chart, where you may see dynamically generated *Actions*. You can click *Expand Actions* to see more details.

Expanded action

In these *Actions*, we suggest things like: * specific PRs that need unblocking * a question to raise in a retro to get more context * someone to check in on during your next one-on-one (*this can only be seen by that person and the people they have 1-on-1s with, per our* [*data ethics principles*](https://www.multitudes.co/blog/measure-what-matters-and-dont-be-creepy)) * something to celebrate with the team When *Actions* are expanded, you can click the 🔗 icon on the headline of the expanded view to share a direct link to each *Action*. ## You can introduce Multitudes without disrupting your team’s workflow Multitudes uses passive data from the collaboration tools your team uses, so once you’ve installed the app, Multitudes pulls out insights while your team works as normal. When you first sign up, we’ll even immediately give you insights based on your historic data! Even better, we also give you updates and notifications within your workflow – via email or Slack. Direct us to [your team’s Slack channel](https://app.multitudes.co/team-settings/notifications) or your email inbox, and we’ll give you updates on things to celebrate and things to watch each week, blocked work that needs attention, people to check in on, and action steps you can take to resolve all of the above. ## **Multitudes helps you become more inclusive** We look at equity and inclusion in practice. Because of systemic bias, good intent sometimes doesn’t translate to fair and equitable actions. That’s why Multitudes looks at behavior, not just intent. For example, we look at [feedback received on PRs](/metrics-and-definitions/people-metrics/collaboration/pr-feedback-received), and check whether anyone is getting less feedback. That’s because [people from marginalized groups get less feedback](https://hbr.org/2016/04/research-vague-feedback-is-holding-women-back) than others. ## **We help you take action in practice** First, our [insights](/metrics-and-definitions/our-approach-to-metrics) highlight changes and outliers in your data. This can help you spot changes in team dynamics, which can be easily missed. Often changes are missed due to unconscious biases, business pressures, or just the usual busy-ness of work life. We know that data is never the full picture, which is why we frame our insights and *Actions* as conversation starters, rather than judgements on how a team is going. For example, in the expanded view of *Actions* example below, we accompany insights with open-ended questions that probe at the human context behind the data.

Take action expanded view

Second, every team is different, so what “good” looks like will also be different. For that reason, we allow teams to [customize targets](/configuration-and-setup/customize-targets). Since our insights are [based on these targets](/metrics-and-definitions/our-approach-to-metrics), customization allows each team to work toward their own goals and to bring their own unique context. As you take action, you can also track your progress in Multitudes – to make sure you live up to your own best intentions. ## **How do I get started?** You can request access and start using Multitudes with a one-month free trial via the [get a demo form](https://www.multitudes.co/demo). Once you’ve been invited, you’ll need someone with GitHub admin access to help with set-up; they can install the Multitudes app in minutes and then we back-pull 6 weeks of historic data. As part of the free trial, we ask that you do regular feedback sessions with us – to improve the app and shape our roadmap. # Setup: Integration Permissions Here’s a quick reference of the permissions required to set up each integration. Check this before set up to make sure everything’s ready to go. *Please note, this is a general guide - your organisation may have custom permissions or settings for some tools which differ to the roles we have described below. We endeavour to keep this as accurate and up to date as possible.* ## Must have before onboarding: Version Control System ### GitHub **Required role:** [Organization Owner](https://docs.github.com/en/organizations/managing-peoples-access-to-your-organization-with-roles/roles-in-an-organization#about-organization-roles)\ **Note:** You must have admin access to select repositories and Github teams/contributors. *** ## Issue Tracking ### Jira **Required role:** [Jira System Administrator](https://confluence.atlassian.com/adminjiraserver/managing-global-permissions-938847142.html)\ **Note:** Non-admin may request the app but this will require an additional approval step from your admin.\ [View Jira integration instructions](/integrations/jira) ### Linear **Required role:** [Admin](mailto:undefined) \ [View Linear integration instructions](/integrations/linear)\ *** ## CI/CD Tooling ### GitHub Actions **Required role:** Any Organisation admin\ [View GitHub Actions integration instructions](/integrations/github-actions) ### Deployments API Anyone with edit permissions in your CI/CD tool can set up the POST request to our endpoint using the token generated in Multitudes\ [View API setup instructions](/integrations/deployments-api) {% hint style="info" %} Note: We currently only allow ***either*** Deployments API or GitHub Actions to be configured. {% endhint %} *** ## Incident management ### PagerDuty **Required role:** [PagerDuty Global Admin or Account Owner](https://support.pagerduty.com/main/docs/user-roles)\ [View PagerDuty integration instructions](/integrations/pagerduty) ### OpsGenie **Required role:** [Account owner or Global Admin ](https://support.atlassian.com/opsgenie/docs/learn-user-roles-and-permissions/)\ **Note:** This is the typical permissions level required to generate an API key in OpsGenie \ [View OpsGenie integration instructions](/integrations/opsgenie)
*** ## Meetings ### Google Calendar **Required role:** [Google Workspace Administrator](https://support.google.com/a/answer/33325?hl=en\&ref_topic=9832445\&sjid=6383127662302688960-NC)\ [View Google Calendar integration instructions](/integrations/google-calendar) ### Outlook Calendar **Required role:** [Privileged Role Administrator](https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/privileged-roles-permissions)\ [View Outlook Calendar integration instructions](/integrations/outlook-calendar) *** ## AI tooling ### Claude Code *For the* [*API (Claude Console) plan*](https://claude.com/pricing#api)*:*\ **Required role:** [Admin role ](https://support.claude.com/en/articles/10186004-claude-console-roles-and-permissions)\ **Note:** You will need to provision an Admin API key (different from a standard API key).\ [View Claude Code integration instructions](https://docs.multitudes.com/integrations/claude-code) *For* [*Teams*](https://claude.com/pricing#team-&-enterprise) *and* [*Individual*](https://claude.com/pricing) *plans:*\ **Required role:** Any\ [View Open Telemetry integration instructions](https://docs.multitudes.com/integrations/open-telemetry) *For* [*Enterprise*](https://claude.com/pricing#team-&-enterprise) *plans:*\ **Required role:** Primary Owner\ [View Claude Code integration instructions](https://docs.multitudes.com/integrations/claude-code) ### Codex **Required role:** Any\ [View Open Telemetry integration instructions](https://docs.multitudes.com/integrations/open-telemetry) ### Cursor **Required role:** [Admin role](https://cursor.com/docs/account/teams/members)\ **Note:** Must be on the [Teams or Enterprise plan](https://cursor.com/pricing).\ [View Cursor integration instructions](https://docs.multitudes.com/integrations/cursor) ### GitHub Copilot **Required role:** [Organization Owner](https://docs.github.com/en/organizations/managing-peoples-access-to-your-organization-with-roles/roles-in-an-organization#about-organization-roles)\ **Note:** You must have admin access to select repositories and Github teams/contributors.\ [View GitHub Copilot integration instructions](https://docs.multitudes.com/integrations/github-copilot) *** ## Notifications ### Slack **Required role:** *May require Slack Workspace Owner*\ **Note:** Different organisations have different permissions required to install apps to a Slack workspace. If your workspace requires "[app approval](https://slack.com/intl/en-nz/help/articles/222386767-Manage-app-approval-for-your-workspace)" you will need a Slack Workspace Owner to install or approve the Multitudes integration.\ [View Slack integration instructions](/integrations/slack)
*** Have any questions about your setup or integrations? Contact us at # Permissions and roles Learn about permission roles and data inclusion rules in Multitudes. ## "Data Inclusion" versus "Permissions" **Data inclusion** controls whose data you see. * If you’re a **Viewer**, your data is **not** included in our insights but you do have access to Multitudes. This is commonly used by managers who are not very active on GitHub, but would still like to view their team’s data. * If you’re a **Contributor**, your data is included in our insights. * Billing is determined by the number of contributors, i.e. the number of people whose data we are processing and showing. ‍**Permissions** controls who can log in, view, and edit things in Multitudes. Here is a table of which permission roles can do what. ### Permission roles 🔐 Note that users with the `No Access` role cannot log in and therefore cannot do any of these actions
ActionMemberManagerOwner
Login and view insights✅✅✅
View their own 1:1✅✅✅
Edit team settings✅✅✅
Install notifications
(e.g., Slack) 		✅✅✅

Install integrations

(e.g., Jira, Linear)

❌✅✅
Add direct reports & view their 1:1s❌✅✅
Update Organization config
(e.g. setting default percentiles)		❌✅✅
Edit other users permission level❌❌✅
Edit billing❌❌✅
{% hint style="warning" %} ‍We encourage all Contributors to have at least some access permissions (Member, Manager, or Owner). This is so that the people whose data is being shared also have access to their own data. Ideally, no one should have “No Access”. This option is available to allow managers to get set up and familiar with Multitudes before inviting their team in. {% endhint %} ### Here are our recommended settings:
Data inclusionNo accessMemberManagerOwner
Viewer
Data not included		🚫 N/A - by definition, Viewers are always able to "view" the appUncommon✅ Managers who are not active on Github✅ Managers who are not active on Github, and want to be able to edit permissions and billing
Contributor
Data included		⚠️ People should only be in this category during setup, when a manager first adds team members to Multitudes. The manager should then invite everyone via email to log in by changing their permissions from "No access" to "Member"✅ Team members✅ Managers who are active on Github✅ Managers who are active on Github, and want to be able to edit permissions and billing
## **How do I switch between Viewer and Contributor?** Instructions to change a team's Data Inclusion status: ### **Viewer to Contributor** 1. Deactivate the Viewer team member in Multitudes by going to their profile in [Settings](https://app.multitudes.co/team-settings/team-member), scrolling to the bottom, and clicking the red `Deactivate` button. 2. Add the team member to your organization on GitHub. With GitHub Teams sync turned on, this should create the user in Multitudes. 3. Give this new user login access by finding them in the [Team Members table](https://app.multitudes.co/team-settings/team-member), and clicking the `No Access` dropdown in the Permissions column. Select a different permission to re-invite them into the app. ### **Contributor to Viewer** 1. Remove the team member from all team(s) in GitHub. With GitHub Teams sync turned on, this should remove that team member from Multitudes too. 2. Invite the team member into Multitudes by clicking the `Invite users +` button on the [Settings > Team Members](https://app.multitudes.co/team-settings/team-member) page. ## Watcher role Anyone with access to Multitudes (e.g., either Contributors or Viewers) and any permission role (e.g., Member, Manager, or Owner) is automatically [getting notifications](/configuration-and-setup/notifications-configuration) for the teams that they are on. The Watcher role simply allows users to get notifications for additional teams that they’re not on. To edit, simply go to the [Settings > Teams](https://app.multitudes.co/team-settings/team) page. The eye icons at the far right of each team row control whether or not the user is watching that team.
Screenshot of Settings > Teams page
# Adding Users & Teams How to add teams and team members in Multitudes You can add your teams and team members automatically via [GitHub Teams](https://docs.github.com/en/organizations/organizing-members-into-teams/about-teams) sync, or manually add your team in our settings page.
Github Teams Sync/pages/lr9BNdyeJCmQYpftLSfC#automatically-adding-team-members-via-github-teams-sync
Manually adding a team/pages/lr9BNdyeJCmQYpftLSfC#manually-add-a-team
Manually adding a team member/pages/lr9BNdyeJCmQYpftLSfC#manually-add-a-team-member
## Enforcing a login option Multitudes allows for login via Google, GitHub, Microsoft, or email and password. To enforce one of these options for your organization, contact . {% hint style="warning" %} Login is enforced based on your organization domain (e.g., @multitudes.com). If you invite a user to your organization via an external email address (e.g., @gmail.com), we won’t be able to enforce their login option. {% endhint %} ## How to give team members login access to Multitudes Login access is a question of permissions. While this is not related to [data inclusion](/configuration-and-setup/permissions-and-roles) (i.e., either Viewers or Contributors can be given login access), it does depend on it! To provide access for a... * **Viewer** - go through the flow to [manually add a new team member](#manually-add-a-team-member) (steps just below) * **Contributor** - you had the option to provide login access when [manually adding a new team member](#manually-add-a-team-member) as a Contributor if you filled in the email address when prompted. If you did not include an email, you can change it from the [Settings > Team members](https://app.multitudes.co/team-settings/team-member). (See image below) The new Contributor that was added will start with a **No access** flag under their name. Go to the Permissions column, click the arrow, and then select a different permission (e.g., **Member**, **Manager**, or **Owner**). After you do this, you’ll see an **Invite Pending** flag under their name. ![Screenshot of Settings Team Member table, highlighting the location of the "No access" label and permission setting.](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/654289b68dce15a9bc6c8f39_4.png) In either case (i.e., once you’ve manually added a new Viewer or Contributor with email, or edited an existing person’s permission to something with access), the person will receive an invite email, and can just click the link in that email to login. {% hint style="warning" %} The invite email will expire after 7 days. You can resend the invite by going to [Settings > Team members](https://app.multitudes.co/teamSettings/teamMember), finding that team member, and clicking the `...` menu on the right of their row, and selecting “Resend Invite”. {% endhint %} ## Automatically adding team members via **GitHub Teams sync** For folks who use [GitHub Teams](https://docs.github.com/en/organizations/organizing-members-into-teams/about-teams), you can automatically keep your teams on Multitudes in sync. This automatically brings in team members, and mirrors the team structure you've set up in GitHub. A few important notes when setting up the Github Team sync: * If you already have teams set up in Multitudes, and then turn on the sync, this will replace your existing teams configuration. * Editing teams is disabled when the sync is turned on. You will not be able to go into a specific team to add or remove team members. This should all be handled in GitHub so you have a single source of truth for team structures. * Anyone that the sync brings in will show up in Multitudes as a Contributor. This may impact [billing](/account-management/billing-and-payments). Since the purpose of GitHub Teams Sync is to keep your teams data in sync with your team structure in GitHub, these contributors can not be turned into Viewers to be excluded from the data (manual tweaking of the teams would defeat the purpose of the sync). * The sync will not bring in someone who previously was deactivated. If a team member was previously on Multitudes, and you deactivated them from their individual team member settings, we will not make them re-appear in Multitudes even if they get updated in your GitHub Teams. * The sync will not bring in someone who was previously a [Viewer](/configuration-and-setup/permissions-and-roles). It will not convert Viewers to Contributors, but it will give them the [Watcher](/configuration-and-setup/permissions-and-roles#watcher-role) role for the team(s) where they have data. E.g. Let's say Pat is a Viewer. The Multitudes app runs a GitHub Teams sync, and finds out that Pat on Team Security and Team Platform in the organization's GitHub Teams. Rather than turning Pat into a Contributor, the sync will simply make Pat a Watcher on those 2 teams and keep him as a Viewer. ### **Follow the below instructions to set-up the sync for the first time:** 1. On the Multitudes app, go to [Settings > Teams](https://app.multitudes.co/team-settings/team) 2. Click the button “Sync with GitHub Teams”

Sync with GitHub Teams

3. On the next page, you will be shown a list of your teams set-up in GitHub. First, select which you want to keep synced. Second, click "Continue". \ \ A few things to consider as you choose teams: * We will only sync the teams that you check in this step. Unchecked teams are excluded from Multitudes. However, if a parent team is checked but its sub-teams are not, members of those sub-teams will still be included as part of the parent team. * Keeping a team un-checked and un-synced does not mean you can then manually configure that team! The automatic sync entirely replaces your existing team configuration; the idea is that GitHub becomes the single source of truth for teams. These selectors just scope which subset of data from your GitHub Teams you want to see in Multitudes. * You can also check “Auto-sync new GitHub Teams from now on” at the bottom of the list. This means, “from now on” any new teams created in Github after this set-up will be added (i.e., any individual teams you un-checked on this page will not be automatically included later, even if this last option to “Auto-sync new GitHub teams from now on” is checked) * When a team is synced and you delete that team on GitHub, we’ll also automatically remove that team in Multitudes as well. Note that team members will not be removed when their team is deleted, unless they are in no other teams after the team deletion.
4. On the next page, we’ll show you if this first set-up sync resulted in any new or removed Contributors, and therefore the [billing](/account-management/billing-and-payments) impact, as well as if it resulted in any changes to your [Linear](/integrations/linear) or [Jira](/integrations/jira) integrations. Click the "Confirm" button to proceed.
5. Once finished, at the top of the Teams page, you’ll see a checkbox indicating that you’re successfully synced with GitHub (you can also un-check to stop syncing entirely, across all teams)! Once sync is set-up, this is what will happen: * Changes to your team should be reflected in \~5 minutes * People with [Owner permissions](/configuration-and-setup/permissions-and-roles) will receive an email: (1) After the initial sync set-up and (2) On an ongoing basis, whenever we detect changes that result in new or removed Contributors * This page will only show you the teams that are synced. Any teams not synced can be added to the sync by clicking the prompt at the bottom
With sync turned on, you can not convert between Contributors ↔︎ Viewers manually via the Multitudes app. This can be managed in GitHub. If you have any issues, don't hesitate to contact us at `support@multitudes.com`. ## **Manually add a team** 1. On the Multitudes app, go to [Settings > Teams](https://app.multitudes.co/team-settings/team) 2. Click the button “Add team +”
3. In the resulting pop-up: 1. Name your team 2. Select team members from GitHub to add to this team (you're limited to selecting team members who are already on our app; to manually add a team member to the app, see the next section) 3. For team members already on our app with [Manager or Owner permissions](/configuration-and-setup/permissions-and-roles), we will detect this automatically, and you can create 1:1 relationships here as well (don’t worry you can always create these later in the My 1:1s page of the application) 4. Click to "Confirm"
## **Manually add a team member** 1. On the Multitudes app, go to [Settings > Team members](https://app.multitudes.co/team-settings/team-member) 2. Click on the “Invite users +” button (part 1) and choose whether you’d like to add them as a **Contributor** or **Viewer** (part 2), from a [data inclusion](/configuration-and-setup/permissions-and-roles) perspective
3. In the resulting pop-up, if you’d earlier selected to add: * **Viewers** (not shown) - simply add people’s email addresses * **Contributors** (image shown below) - we show you a list of contributors from your connected GitHub that aren’t currently in the Multitudes app. Once you check someone’s GitHub username, an email field will show up at right. If you leave this blank, their data will be incorporated into our insights, but they themselves will not be able to access the app, i.e., **No Access** [permissions](/configuration-and-setup/permissions-and-roles). If you fill this in with their email, they will be sent an invite to access the app with default **Member** permissions (you can change permissions later from the [Settings > Team members](https://app.multitudes.co/team-settings/team-member) page to Manager or Owner permissions).
{% hint style="info" %} If you’re adding **Contributors**, clicking the “Select contributors” button in the blue banner at the top of the page (which appears whenever we detect new GitHub contributors not yet on Multitudes, see example in the screenshot for Step 1 above) will directly take you to the same Step 3 as if you’d clicked “Invite users +” and then selected **Contributors**. {% endhint %} ## Backdating a team member's team history Sometimes the date a team member is added in Multitudes doesn't match when they actually joined the team. For example, if someone is manually added a few weeks after the Multitudes team was created, but they'd actually been part of it since day one. In cases like this, you can backdate their team history so it accurately reflects their real start date, rather than the date they were added in Multitudes. To request this change, contact us at with: * The team member's name * The team name * Their correct start date ## Deactivating a team member {% hint style="info" %} **Only Owners can deactivate team members.** {% endhint %} When you deactivate a team member, Multitudes will no longer show data for this person from this point onwards and the team member will lose access to Multitudes (if they currently have access). To do this, go to the [Settings > Team Members](https://app.multitudes.co/team-settings) and look up the team member you want to deactivate. Go to their profile and at the bottom, click the Deactivate button. Then when the modal pops up, click "Deactivate `{name}`" to confirm.
#### Expired invite If the user has an expired invite, you will need to cancel their invite first before you can deactivate. To do this, find the team member in the [Settings > Team Members page](https://app.multitudes.co/team-settings). Click the 3 dots at the end and cancel invite. Once this is done, you can deactivate per instructions above.\ \ If the user was invited as a viewer, cancelling their invite will automatically deactivate them.
#### View deactivated users and reactivate them You can view deactivated team members by setting the "Status" filter to "Deactivated" in the Team Members page. To reactivate a team member click the three dots in their row and then click "Reactivate". {% hint style="warning" %} **If you have Github Teams Sync enabled:** * Team members will automatically be deactivated when they are removed from all synced teams in Github. * You cannot currently reactivate your team members via Multitudes. Please contact for help. {% endhint %} # Configuring your Team Manage team assignments and configure contributor levels for effective team organization. ## People who work on multiple teams or no specific teams We’ve left the team structure flexible to meet your needs: We allow people to be on multiple teams, or on no teams at all. ## Configuring levels You can configure the seniority levels of team members who are [Contributors](/configuration-and-setup/permissions-and-roles) to provide color-coding on the [Feedback flows](/metrics-and-definitions/people-metrics/collaboration/feedback-flows) chart. At the moment, anyone can edit the level for anyone else. To do so, go to [Settings > Team members](https://app.multitudes.co/team-settings/team-member), click on a Team member who is a Contributor (see the column on Data Inclusion). On the page that opens for that team member’s profile: 1. Find the section for Level and edit the drop-down 2. Then click ‘Confirm’ at top right to save
Team member profile page with highlight showing the level dropdown and then confirm button.

Edit level

### Customizing Level options {% hint style="info" %} Only Owners and Managers can customize level options. {% endhint %} You can edit the Level options available to select by going to your [Organization Settings](https://app.multitudes.co/team-settings/organization-settings?section=customize-levels) page (only accessible to Owners and Managers). You have up to six levels available and can rename these to match your organisation's conventions. Your level titles will be visible in the above profiles, as well as on the [Feedback Flows](/metrics-and-definitions/people-metrics/collaboration/feedback-flows) chart.
App page with heading Customize levels and 6 fields to enter new level titles.
# User Linking Link contributors across integration accounts to ensure accurate data attribution. Multitudes contributors must be linked to their corresponding user accounts from your integrations in order for data from that integration to appear in the app. This is so that Multitudes knows whose data is whose, and so that contributors can have confidence that their data is being attributed accurately. For example, if you link the contributor `Sam Smith` in Multitudes with the user `sam-smith` in Jira, that tells us that the issues assigned to `sam-smith` should be attributed to the `Sam Smith` and the teams they're part of in Multitudes. Viewers, by definition, don't show up in our data, so if you want to see someone's data, you need to switch them to a contributor. [Learn more about data inclusion settings](/configuration-and-setup/permissions-and-roles). {% hint style="warning" %} Currently Multitudes requires that all contributors are GitHub users. {% endhint %} Currently, to be a Multitudes contributor, someone must be a GitHub contributor. If they Jira or Linear, for example, but not GitHub, they can not become a contributor, and therefore can't have their data shown in Multitudes. ## Linked vs. Verified
Table of different integration tag states depending on linking & verification state.
There are 3 states that a user can have: * **✅ Linked & verified:** This Multitudes contributor has been linked to a user on this integration, and this link has been verified via email. These links can not be edited. If the Multitudes contributor has login access, and the email they used for Multitudes is the same as the email they use for the given integration, they will automatically be linked & verified with no email or any further action required. * **⚠️ Linked but not verified:** They have been linked to a user, but they have not yet confirmed via email verification that they are indeed this user (see below for more info). This is to stop data from being attributed to the wrong people. You can click these links to send (or re-send) a verification email, which the user in question will need to accept to confirm that they are indeed this person. * **❌ Unlinked:** The Multitudes contributor is not linked to a user on this integration. Their data fom this integration will not be shown in the Multitudes app, since we don't know who to attribute this data to. {% hint style="warning" %} We can't verify user links with Jira yet. Our UI will show links to Jira users as linked with no further action required, but on hover you will see a tooltip that indicates that they are not verified and that verification is not available. {% endhint %} ## Email verification If verification is required (e.g. the contributor has access to Multitudes, and the email they use for Multitudes is different to the email they use for the integration in question), we send an email to the user's integration email address. The user can then click the magic link from their email, which lets us know that they are indeed the same person. If they are linked to users in other integrations, and they also use the same email address, those links will automatically be verified. *Example: The Multitudes contributor "Sam Smith" logs in to Multitudes using `sam.smith@acme.org`. They have the following user links:* * ✅ *"s.smith" on Jira which uses `sam.smith@acme.org`. This gets verified automatically, because it's the same as their Multitudes email address.* * ⚠️ *"sam-smith" on Linear and ⚠️"Sammy Smith" on PagerDuty which both use an aliased work email `sam@acme.org`. Multitudes sends a verification email to this address. Clicking the link in the email will verify both Linear and PagerDuty.* * ⚠️ *"sammy-the-coder" on GitHub which uses their personal email `sam.loves.dogs@gmail.com`. This will need a separate verification email as it's a different address* ## Editing user links in bulk from integration settings Go to [Settings > Integrations](https://app.multitudes.co/team-settings/integrations). Click "Configure" on the integration that you'd like to edit the user links for. This will bring up a modal listing all your Multitudes contributors. You can use the dropdown next to each contributor to select the corresponding user in the integration.
## Editing user links for a specific contributor You can link an unlinked contributor, or change the link of an already-linked contributor, in one of these three places. Unless you are a Manager or Owner, you can only edit user links for your own team member. 1\. [Settings > Team Members](https://app.multitudes.co/team-settings): Click on a specific team member. Under their display name, you will see a row of tags showing the integrations you have installed. Clicking on these will open a modal for editing that link. 2\. [My 1:1s](https://app.multitudes.co/1on1-guides): Click on a specific team member's 1:1. Under their display name, you will see a row of tags showing the integrations you have installed. Clicking on these will open a modal for editing that link. 3\. [Settings > Teams](https://app.multitudes.co/team-settings/team): Click "Edit" on a team. There is table of user links, with contributor names & profile images, and the logos of installed integrations. You can click the checkboxes/icons to open a modal for editing that link. # Configuring Working Hours Customize and back-date working hours and non-working days for accurate out-of-hours work calculations. ## **“Exclude weekend hours” toggle** On some Trend Summary cards on our My Insights page, and on the [Flow of Work](https://app.multitudes.co/flow-of-work) > filters bar, there is an `Exclude weekend hours` toggle. You can turn this toggle on to exclude weekend hours from `Change Lead Time` and its subsets (`Coding Time`, `Review Wait Time`, `Editing Time`, and `Deploy Time`). “Weekend hours” are based on the individual's configuration in [Settings](https://app.multitudes.co/team-settings), and it considers all non-working days with 24 hours excluded for each non-working day. For people who work a typical Monday-Friday workweek, this toggle will just exclude weekends (Saturdays and Sunday, 48 hours total). If someone's configuration shows that they don't work Fridays, then this toggle would exclude Fridays-Sundays from the calculation for PRs this person authored (72 hours total). To update this, go to [Settings](https://app.multitudes.co/team-settings), search for the individual you want to update, then click into their profile and look for "Working Times". ## Changing standard working hours Multitudes uses people’s working hours to calculate [out-of-hours work](/metrics-and-definitions/people-metrics/wellbeing/out-of-hours-work). We have a default setting for this (Mon-Fri, 8 am - 6 pm in the timezone of your company’s headquarters), but because people work flexibly, we strongly recommend that you configure this for your individual preferences.To change the working hours for you or a teammate, go to [Settings > Team Members](https://app.multitudes.co/team-settings), and select the person you want to adjust. Scroll down to the “Working Times” section, make changes as needed to the Timezone, Work days, or Working hours, and then choose “Confirm.” ## Back-dating a team member's working hours Some insights on Multitudes, such as out-of-hours work, are based on a team member’s working times. These are individually configurable but we start with the following defaults: * Timezone: The organization’s timezone * Working days: Monday to Friday * Working hours: 8 am to 6 pm If you have a team member who works different hours or in a different timezone, you will likely want to update the default settings and back-date the changes so that person's historical data matches their actual working hours. To do that, follow the steps below. First, go to [Settings > Team Members](https://app.multitudes.co/team-settings), and select the person you want to adjust. Scroll down to the “Working Times” section and make changes as needed to the Timezone, Work days, or Working hours. After you make changes to the "Working Times" section, a banner will appear (see screenshot below). Choose “Yes, back-date these changes”.
Screenshot of the working times settings in the Multitudes app, with a banner asking whether to back-date changes.
This will back-date your current working time settings, applying the current settings to the last 12 months of data for this person. This includes recalculating metrics that use these settings, such as Out-of-Hours work. {% hint style="warning" %} If you do **not** want to back-date the changes to Working Times, click “No, update from today” and the app will apply your settings from that point onwards, just like any other setting. {% endhint %} # Customize Work Categories How to customize 'Types of Work' and 'Feature vs Maintenance' charts (for Jira and Linear) Each organization works differently. Our custom configuration allows you to accurately understand *what work has been completed* at your organization, based on issue tracking data from integrations like Jira and Linear. We hope this will help keep teams on track and prioritize work. ## Work “completed” definition {% tabs %} {% tab title="For a Jira integration" %} Work completed = work that has been moved to the `Done` status, or to a custom status in the `Done` status category. Use the toggle on the chart to show the sum of issues or total story points. For example: if you have custom Jira statuses like `Testing`, `In Staging`, `Ready for Release`, and `Released`, and the last 2 statuses are both in the `Done` category, then we count number of issues moved either to `Ready for Release` or `Released`. Note: We exclude subtasks from the analyses. {% endtab %} {% tab title="For a Linear integration" %} Work completed = work that has been moved to `Done`. Use the toggle on the chart to show the sum of issues or total story points. We include both issues *and* sub-issues in our sums. {% endtab %} {% endtabs %} ## How to customize “work” completed {% tabs %} {% tab title="For a Jira integration" %} Work can be defined by using [Issue Type](https://community.atlassian.com/t5/Jira-articles/Understanding-issue-types-in-jira/ba-p/1497237), [Epic,](https://support.atlassian.com/jira-software-cloud/docs/what-is-an-epic/) and/or [Label](https://community.atlassian.com/t5/Jira-articles/Using-labels-in-Jira/ba-p/1782833) (you can not select Subtask or issue types of [custom hierarchy levels](https://support.atlassian.com/jira-software-cloud/docs/configure-custom-hierarchy-levels-in-advanced-roadmaps/)). We will only show epics that are currently in an “open”-type status. * If you define a category using an epic that later gets closed, the chart will still show the issues in that closed epic for that category, until you decide to remove this epic from the category definition. * If you remove the closed epic from your configuration, it will disappear from the epics dropdown and you will no longer be able to use it. Our default for `Types of Work` is to categorize issues by `Issue Type`. The Jira defaults for this are `Story`, `Bug`, or `Task`, but you may have your own custom issue types. Our default for `Feature vs Maintenance` is to categorize issues by `Issue Type`, where issues called bug, tech debt, or chore are displayed in various shades of purple, to indicate that it’s all `Maintenance work`. All other issues are grouped into the green `Feature work.` Note that an older release of this chart defaulted to categorizing issues based on whether it *contained* a string called `bug`, `tech debt`, or `chore`. Now, the config default will look for an exact match (of course, configurable!). {% endtab %} {% tab title="For Linear integration" %} Work can be defined by using [Project](https://linear.app/docs/projects) and/or [Label.](https://linear.app/docs/labels) Our default for `Types of Work` is to categorize by `Project`. Our default for `Feature vs Maintenance` is to categorize by `Project`, where projects called bug, tech debt, or chore are displayed in various shades of purple, to indicate that it’s all `Maintenance work`. All other projects are grouped into the green `Feature work`. You can define categories of work based on combinations of `OR` and `AND` conditions. For example, you can define a custom work category, `Customer Support`, that is defined as where an Epic is `Support`, and the Label is one of `Customer Apple`, `Customer Grape`, `Customer Peach`. Configure these categories based on how your teams think about where your time goes. This will help to see at-a-glance where your team’s split of time is what you would expect. Note that we run our Linear pipeline hourly, so changes you make to issues in Linear will come through in an hour or less. {% endtab %} {% endtabs %} ## How "work" is defined in custom configurations * Each chart’s configurations are organization-wide. * For now, we do allow categories to overlap: * If you define a custom work category called `Support` which includes work where Label is `Customer`, and another custom work category `Customer` which also includes work where Label is `Customer`, then work completed with a Label that is `Customer` will be counted in both categories. * This helps some companies that prefer to have an accurate relative comparison of work across categories. * In the Feature vs Maintenance chart only, we will flag this discrepancy because it means some issues are being double counted, and it inflates the total issues done for that week or month (to over 100%). * For now, we handle work that falls outside custom categorization differently across the two charts: * In the `Types of Work` chart, anything that is not captured by custom categories will go into the `Unassigned` bucket (which cannot be removed) * In the `Feature vs Maintenance` chart, you have 2 options. Work that falls outside configured rules can either be: * Counted as `Feature work`, by selecting the “Everything else” option when editing the Feature category (see image below; this is the default - since a lot of teams are using this chart to get a quick high level view of the two buckets, it’s easier to think of all work that’s not Maintenance as part of the Feature) * Not counted at all, by selecting the “Only specific issues” option when editing the Feature category (see image below; this means some work will be completely excluded from the chart if it isn’t captured by the defined conditions)

Edit feature category

# Notifications Configuration Set up and customize Slack and email notifications to receive timely insights about your team. ## **How to configure n**otifications ### Email Currently, only the Trend Summary notification is available by email, and it cannot be configured. If you’d like to unsubscribe, you can click the unsubscribe link in the email, or email us at ### Slack Once our Slack integration has been set up, the[ Settings > Notifications](https://app.multitudes.co/team-settings/notifications) page will show various configurations. * To set up a new notification (unchecked box), check the box to show the config modal with steps to configure (e.g. channel, frequency). * To edit a notification, click the link text next to the checked box to bring up the config modal. * To turn off a notification, uncheck the box. ![Screenshot](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/64ee67e00da86184dbf36305_image%20\(1\).png) You can also edit specific notifications directly by clicking the bell icons on the My Insights page. * The bell icon next to the `Trend Summary` section headline links to the [Trend Summary notification](/knowledge-base/types-of-notifications/trend-summary) * The bell icon on the cards for Change Lead Time, Merge Frequency, and Change Failure Rate, in the `Take Action` section link to the [Daily Blocked PRs notification](/knowledge-base/types-of-notifications/work-digest). Clicking the bell icon will either prompt you to connect to Slack if you have not set up our Slack integration, or show a modal for configuring the relevant notification if you are already set up. ![Screenshot](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/64caa792888c778fe0fbe501_homepage%20for%20alerts.png) ## Which teams do I get notifications for? It depends on your team settings and permissions. For both the [Trend Summary email](/knowledge-base/types-of-notifications/trend-summary) and [Multitudes AI Coach Slack](/knowledge-base/types-of-notifications/multitudes-ai-coach) notifications, whether you receive these notifications at the team - or company - level depends on set up and [permissions](/configuration-and-setup/permissions-and-roles), as well as whether you’re either part of a team or [watching](/configuration-and-setup/permissions-and-roles#watcher-role) it, both of which are configurable on the [Settings > Teams](https://app.multitudes.co/team-settings/team) page. **If your company has at least one team set up** * You're a team member or watcher: you will receive notifications for *each* team that you're on or watching (permissions don't matter) * You have Manager or Owner permissions: you will receive company-wide notifications * You have Member permission but you're not on a team or watching a team: you will *not* receive any notification **If your company does not have teams set up** * You will receive the notification for the company as a whole (permissions don't matter) ## Explore our Types of Notification
PR Updates – ad hoc/pages/odB5zJD1A6dozBEdVjGz
Work Digest – daily/pages/izWYNFa7lVE7O6aekHRf
Trend Summary – weekly/pages/olXjslxCTaD0S8iYg2e4
1:1 Prompts/pages/DiwQ5SsvM51AP5rJXUeP
Annotations/pages/TwThXenFIHXqP69ObqPl
# Customize Targets Set custom performance targets for your metrics. To give an idea of what “good” looks like, our metrics show a default industry benchmark as default targets. You can view each of these benchmarks in the "What good looks like" section of each metric's documentation. We also allow teams to set custom targets, because we recognize that teams may have different considerations (e.g., hardware teams may be limited in `Change Lead Time` due to physical operations, some teams may want to beat industry standards). {% hint style="warning" %} Note: When you set a custom target, it becomes the new target for everyone viewing that metric on that page—custom targets are shared across your organization, not personal preferences. {% endhint %} Targets, whether custom or industry benchmarks, are what drive the insights that Multitudes shows you, whether in app or via our notifications in Slack or email. There are two ways to customize targets: 1. **Settings > Targets page**
Navigate to [Settings > Targets](https://app.multitudes.co/team-settings/targets) or click the “Customize targets” button on the top right of the *At a glance* section on the My Insights homepage. Customizing a target is as simple as clicking on a target and editing it! To be clear, in this table: * Rows represent teams (including a row for the organization as a whole at the top) * Columns represent the metrics * Each cell represents a target For a cell with a given team row and metric column, if the text of the cell is colored: * **Black**, this means the current target is unchanged from the industry benchmark (the industry benchmark is shown on the first gray row, for reference) * **Teal**, this means a custom target has been set {% hint style="info" %} Note that this table currently lets you set targets for 5 of our metrics, `Change Lead Time`, `Merge Frequency`, `Change Failure Rate`, `Out-of-Hours Work`, and `PR Participation Ga`p. To set targets on other metrics (or also for these metrics, as a secondary option), see the option below. {% endhint %}
2. **From a chart legend** On a metric-specific pages (as accessed from the left hand menu bar), wherever you see a chart with a green target zone, you can customize the target here by going to the legend and clicking on the small target icon next to the “Target” label. This will show a modal from which you can edit the target. Click “Save Changes” to trigger a chart refresh. {% hint style="warning" %} If you’ve selected multiple teams or have “All” as the team filter at the top right of the page, the target shown in the legend will be the target for the organization as a whole. {% endhint %}
# Single Sign On (SSO) We offer SSO through Okta. To enable this, you'll need to be on our Enterprise plan; reach out to to kick this off. We'll give you a connection name that you'll need during the set up process. {% hint style="info" %} You'll need the connection name from Multitudes for SSO setup, so don't continue until you have that. {% endhint %} First, create a new application in Okta. For this, an Okta administrator will need to: 1. **Create a new app** 1. Choose OIDC as the sign-in method, and Web Application as the application type. 2. Choose Next.
2. **Add information about New Web App Integration** 1. Name the application `Multitudes` 2. Set the sign-in redirect URI to `https://multitudes.us.auth0.com/login/callback` 3. Set the sign-out redirect URI to `https://app.multitudes.co/login` 4. Set the trusted origins to `https://multitudes.us.auth0.com` 5. Select the `Skip group assignments for now` radio button 6. Hit Save; the application will be created
3. **Edit the General Settings section of the new application:** 1. Scroll down to General Settings and choose Edit.
4. **Edit Login Settings** 1. Under Login, change the drop-down at `Login initiated by` to `Either Okta or App`. 2. Tick the `Display application icon to users` checkbox. 3. Set the `Initiate login URI` to `https://app.multitudes.co/login/social?connectionType=okta-your-organisation`, replacing `okta-your-organisation` with the connection name given to you by Multitudes. 4. Leave the `EMAIL VERIFICATION EXPERIENCE` section blank
5. **Tell Multitudes the app details** 1. We need to know the `Client ID` and `Client Secret`. These will be used to set up your Okta integration on our side, and are required for secure authentication of your users. 2. Copy your Okta domain — this can be found in the menu at top right of your Okta browser tab. 3. Use 1Password or a similar tool to securely transfer the `Client ID`, `Client Secret`, and Okta domain to Multitudes. # API Keys Users can generate a variety of API keys for interacting with Multitudes via API. ## Types of keys API keys come in 2 types: Personal API keys and Organization API keys. #### Personal API keys These keys are used when sending [OTel metrics](/integrations/open-telemetry) from tools like Claude Code to the Multitudes OTel collector. They are personal to each engineer, so they pose less risk if compromised. #### Organization API keys These keys are for performing activities on behalf of an organization, including: posting deployment metrics to the Deployments API, querying the Multitudes API, or sending OTel data from a central collector to Multitudes (rather than directly from an engineer's machine). ## Permissions needed to generate a key
Key typeScopesPurposePermissions to generate
Personal Open Telemetry: writeSending raw OTel data from an individual's machine direct to MultitudesMembers, Managers, and Owners
OrganizationOpen Telemetry: writeSending aggregated OTel data from multiple engineers machines to Multitudes via a centralized collector Managers and Owners
OrganizationDeployments: writeSending deployments to the Deployments APIManagers and Owners
OrganizationData: readQuerying the Multitudes API Members, Managers, and Owners
## Generating an API Key 1. From the [Settings > API Keys](https://app.multitudes.co/team-settings/api-keys) page, click "+ Generate New Key" on the top right corner of the page. This will pop-up a modal.
2. In the modal, select your key type. Choose a Personal or Organization key based on your usecase. 3. Select the appropriate scopes for your use case. Scopes can be set to no access, read, or write 4. Click "Create key".
4. After the key has been generated, copy it for use. The full key will only be shown once.
# Multitudes Insights Learn about the insights Multitudes provides and how these are calculated. In addition to helping you visualize key metrics, we also share insights about those metrics. Insights are made up of the following: * **Insight text:** The message at the top of a chart, describing the metric’s value and trend * **Insight value:** Where the metric is at currently * **Insight trend:** Whether the overall trend over the date range has been up, down, or flat over the selected date range. This is indicated by the arrow. The arrow color changes depending on whether the trend is towards the target or away from it (for example, an upward trend for Merge Frequency will be green because more merges are generally better) * **Insight status:** A combination of the insight value, trend, and the metric’s target. This is represented by thumb icons (see the top right corner of each metric card in the *Trend Summary,* or throughout the *At a glance* section): * ![red thumbs down icon](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/642b2c2de7085846f7f7ad9c_red%20thumb.svg) The insight value is outside of the target and not getting closer to it * ![yellow sideway thumbs icon](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/642b2c9307a4639376950a04_thumb-neutral.svg) The insight value is outside of the target zone but trending towards it, or within the target zone but close to the threshold * ![green thumbs up icon](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/642b2c9e9334b2eeb423051b_thumb-up.svg) The insight value is well within the target Any changes to custom targets will immediately affect the insights (text, value, trend, and status) and *Actions* in the app. Of course, the time range of data will also affect insights and *Actions*, in this case, it's important to know that the *At a glance* view and *Trend Summary* behave differently. ### At a glance view
As this view is designed to be a high-level overview, the date range is fixed to the last 6 weeks. The insights and *Actions* are only affected by changing custom targets. In other words, changes to selected dates in the *Trend Summary* section will not affect this view. ### Trend Summary
As this view is designed to help teams take action with granular data, the date range can be modified. The insights and *Actions* are therefore affected by both changing dates and changing custom targets. {% hint style="info" %} **Note:** We use the organization timezone for most dates, including the date selector, the times used to show datapoints on charts, and in the drilldowns. Sometimes we will *also* show local timezones in the drilldown, e.g., on our Wellbeing page, so you can see the impact of an event on the person. {% endhint %} # Our Approach to Metrics We measure what matters We’ve spent a lot of time thinking about indicators of team success. Everything we measure in our tool is based on conversations with experts on engineering management, research on how to support DEI at work (diversity, equity, and inclusion), and metrics frameworks like [DORA](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) and [SPACE](https://queue.acm.org/detail.cfm?id=3454124). Our team has worked as developers, data scientists, engineering leaders, and coaches for engineering teams. Equity and inclusion is at the heart of all that we do: our CEO, Lauren Peate, ran a diversity, equity, and inclusion consultancy before starting Multitudes, and our whole team is committed to unlocking the collective power of teams – a key part of which is ensuring that those teams are equitable. Our focus is on how to show the holistic view of team delivery, because productivity is about more than just speed and output. As [the recent paper on SPACE metrics](https://queue.acm.org/detail.cfm?id=3454124) points out, metrics signal what is important to an organization – and flow metrics alone cannot capture critical dimensions like employee satisfaction, well-being, retention, collaboration, and knowledge-sharing. This is why we not only provide all four DORA metrics but also provide people metrics that look at wellbeing and collaboration. **Read on for a deep dive into our metrics – what they are, why they matter, how we measure them, and how you can get the most out of them for your teams.**
Flow of Work/pages/kefB0m7E6aeloFs4t1J4
Value Delivery/pages/ZuU3xuY5xxLJB2q5fC4Y
Quality of Work/pages/CF7b93IG2dSigxJDh4Aw
Wellbeing/pages/j7pAbMu8DN4v2JpJg0w6
Collaboration/pages/Zp3FnJ9VUeiV1Vym10Ab
{% hint style="success" %} **What good looks like**\ \ Our metrics show pre-defined benchmarks based on internal and external research. You can read more about the research behind these benchmarks in each metric section below. That said, because each team is different, we allow teams to [customize targets](/configuration-and-setup/customize-targets). {% endhint %} {% hint style="info" %} Learn more about our Data Ethics Principles in [this blog article](https://www.multitudes.co/blog/measure-what-matters-and-dont-be-creepy). {% endhint %} **To see our metrics in action, try it out for yourself – you can sign up for our beta program** [**here**](https://www.multitudes.co/join-our-beta)**!** # AI Impact Feature Learn about our approach to measuring the impact of AI Use this feature to identify patterns in AI adoption across different user segments and understand how AI coding assistants affect your team's development metrics. ### Get more from your AI tooling – the research-backed way This feature is based on our original research about the impact of AI on engineering teams, where we followed 500+ developers for months. We combined telemetry data from real development workflows, AI tooling telemetry data, comprehensive surveys, and in-depth interviews with engineering leaders and team members. This research helps us understand not just whether AI tools are being adopted, but how they affect productivity, code quality, collaboration patterns, and developer wellbeing. Our findings show that the actions you take as a leader have the biggest impact on the success of your AI rollout – not your tooling. With the right initiatives, you can help your team get more benefit from AI, with fewer of the costs (to codebase quality, learning, and more). But to do that, we need to be able to: * See what AI interventions are working and which aren’t * Support our team members to learn – whether they’re engaged or skeptical * Measure the full impact of AI – we don’t want to focus so much on productivity that we lose sight of code quality or the impact on developer experience * Get early warning signs of AI slop This feature will help you do just that. Follow the links below to learn more. If you're curious about our original research, check out [multitudes.com/research](https://www.multitudes.com/research). ### Getting started To get started, set up integrations with your AI tooling. The installation docs are here: * [Claude Code](/integrations/claude-code) * Codex – we support Codex [via Open Telemetry](/integrations/open-telemetry) * [Cursor](/integrations/cursor) * [Github Copilot](/integrations/github-copilot) We also support [Open Telemetry](/integrations/open-telemetry) - so let us know if there's another OTel data source you'd like to see here!
AI Adoptionhttps://docs.multitudes.com/metrics-and-definitions/ai-impact-feature/ai-adoption
AI Impacthttps://docs.multitudes.com/metrics-and-definitions/ai-impact-feature/ai-impact
# AI Adoption Learn about AI adoption patterns for your team Track how your team is adopting AI coding assistants and visualize usage patterns across different tools, team members, and time periods. AI adoption doesn't happen uniformly—it varies by individual preference, seniority, delivery pressure, and organizational enablement efforts. Our [research](http://multitudes.com/data-to-cut-through-the-hype) shows that simply purchasing AI tools doesn't guarantee adoption. Successful rollouts require deliberate enablement, peer learning, and creating space for experimentation outside of high-pressure delivery cycles. ## Daily Active Users (DAUs)
#### What it is: Daily Active Users (DAU) measures the percentage of your team actively using AI coding assistants each day. When viewing weekly charts, we show the average of daily percentages across that week. As a bonus, we've included DAU insights not just for your Multitudes contributors but for anyone across the organization who's using these AI tools (the non-contributors). That means you'll be able to see how leaders and people outside of engineering are using AI tools. {% hint style="warning" %} Currently, the DAU charts include weekends. If you'd like to set targets to account for weekends, see the [#how-to-update-your-targets](#how-to-update-your-targets "mention") section below. {% endhint %} #### Why it matters: DAU helps you understand adoption momentum and identify patterns over time. A high DAU indicates that AI tools have become part of your team's daily workflow. Sharp increases often follow enablement sessions or peer learning initiatives, while dips during high-pressure periods (like before a product launch) may indicate that developers don't have time to learn a new tool, or don't see the value of AI in making them faster. #### How we calculate it: The DAU percent shows you the number of person-days where someone used AI tooling out of the total number of person-days for that group. For example, if we're looking at the DAU over a week for a team of 10, and 3 people were active on 5 days, 6 were active on 4 days, and 1 was active on 3 days, then we would have: DAU = (Person-days active = 3\*5 + 6\*4 + 1\*3 = 42) / (Total person-days possible = 10\*7 = 70) = 60% Notes: * The group is defined by your filters and view, and we include everyone in the group in the DAU calculation, including active and inactive AI users, and those without AI accounts or integration with Multitudes. In brief: If someone is in the group, we include them in the total person-days possible, whether or not they have AI access. * We include activity on **any** AI tool, and we aggregate across tools (so if someone used 3 AI tools in 1 day, that's 1 person-day active; if someone used 3 tools on 3 consecutive days, that's 3 person-days active). The exception is in the AI Tool tab, where we calculate DAU separately for each specific tool — so if someone is active on both Cursor and Claude Code on the same day, they count towards the DAU for both tools. * A user is considered "active" on an AI tool if their work used tokens, created cost, generated lines added/deleted, or accepted/rejected suggestions. Non-contributors refer to Multitudes users that aren't Multitudes contributors – people whose data isn't normally reflected in Multitudes ([see more about how we define Contributors](https://docs.multitudes.com/configuration-and-setup/permissions-and-roles)). We have included non-contributor insights for free in this and the Intensity of Usage chart (below) because we recognize the importance of understanding AI impact beyond traditional engineering roles. {% hint style="success" %} #### What good looks like In our research, we found that teams with strong AI adoption typically see 50%+ DAU among their high AI users when measured across all five working days. However, in the app we’ve set the benchmark at 35% DAU to account for weekend exclusions in our research methodology (weighting 50% for 5 days a week = 50% \* 5/7 = 35.7%). This adjusted target provides a more appropriate comparison for typical workweek usage patterns. Also note that DAU is the starting point for whether AI could be having an impact in your organization; you’ll need to combine this with the Intensity of Usage metric and our AI Impact Measurement feature to understand outcomes. Look for sustained usage over time rather than initial spikes that fade. {% endhint %} #### How to update your targets: {% hint style="warning" %} Changing the DAU target also changes the threshold for computing High and Low AI users in [AI Impact](/metrics-and-definitions/ai-impact-feature/ai-impact). {% endhint %} 1. Click on the Target text under the Daily Active Users or Impact on Adoption charts. The modal to update the target should appear.
2. Set the target you'd like for your organization by completing the text field or clicking on one of the days per week buttons.
3. Note that DAU data includes weekends, so for full-time weekday usage, it should be 71% (5/7). If you want 3 days a week usage, that's 3/7 = 43%. 4. Click "Save Changes". ## Intensity of Usage
#### What it is: Intensity of Usage complements the DAU metric – while DAU shows how often people are using AI tooling, this chart shows how deeply people are using these tools. We show intensity of usage based on cost, input tokens, and lines changed. Like the DAU chart, we combine activity across all your AI tools. Also like the DAU chart, we include this data not only for Multitudes contributors but for anyone across the organization who's using these AI tools (the non-contributors). #### Why it matters: Intensity of Usage helps you understand how deeply your teams are engaging with AI tooling, not just how often people are logging in. This matters because two teams can have similar DAU, but very different patterns of usage: One may be using AI lightly for occasional assistance, while another may be relying on it heavily as part of everyday work. Looking at intensity alongside DAU helps you identify your AI superusers. #### How we calculate it: We offer three metrics to measure intensity. * **Cost** — The cost of AI tool usage. Note that this is different from spend for subscription-based plans. Instead, it's showing what the LLM reported was the cost of the queries that the user sent. We show this in USD. * **Input tokens** — The number of tokens sent to AI models. We show input tokens rather than output tokens because the input tokens are more linked to user actions. In addition, using reasoning mode can dramatically increase the number of output tokens even if the user inputted the same prompt as someone not working in reasoning mode. * **Lines changed** — The number of AI-suggested lines of code that were changed. We use lines changed (rather than lines suggested) because lines suggested can vary by model, so it makes it harder to compare across models and tools, and because lines changed shows what a human decided was close enough to start with as a base (even if the human then made significant changes later). Not all metrics are available for every AI tool integration. A summary of data availability by AI tool integration is shown below:
MetricClaude Code (API)Claude Code (OTel)Claude Code (Enterprise)CursorCopilotCodex
Lines changed✅✅✅✅✅❌
Input tokens✅✅❌✅❌✅
Cost✅✅❌✅❌❌
For each metric, intensity of usage can be shown either as a **total** for your selected teams or as an average **per person**. The average-per-person view is calculated by taking the total usage for the selected metric (e.g., total lines changed by everyone in the group) and dividing it by the number of contributors in the group (e.g., the people on the selected team). As with the DAU chart above, one of the tabs on this chart allows you to view it for non-Multitudes contributors, people whose data isn't normally reflected in Multitudes ([see more about how we define Contributors](https://docs.multitudes.com/configuration-and-setup/permissions-and-roles)). We have included it in this here because we recognize the importance of understanding AI impact beyond traditional engineering roles. {% hint style="success" %} #### What good looks like We looked at the 75th percentile across our customers to understand what good looks like. We found: * **Lines changed:** 975 lines changed per person per week – the top 25% of AI users accept at least that, or more * **Input tokens:** 1.1 million tokens per person per week * **Cost:** $27 USD per person per week. Note that Copilot doesn't provide token or cost data so they're not part of the token or cost analyses above. We'll refresh these benchmarks over time as the AI landscape evolves. {% endhint %} ## Super-users
List of AI Super-users based on US spend on AI tooling, number of lines accepted and percent of daily active usage.
#### **What it is** Super-users are team members who demonstrate consistently high AI tool usage. These individuals have often developed practices and workflows that work well on your specific codebase. #### Why it matters Super-users are a great source of insights on how to get more from your AI tooling. Super-users are usually high AI users because they enjoy learning the latest AI practices, and they’ve already learned what works best on your organization’s codebase. (For example, AI still struggles with [complex codebases](https://arxiv.org/pdf/2406.17910) – but your AI super-users may have developed unique ways to get around this.) Our [research](http://multitudes.com/data-to-cut-through-the-hype) shows that not only do these users have great principles for using AI well, but their peers really want to learn from them, so super-users are a great place to start with your peer-to-peer learning initiatives. #### How we identify super-users Super-users are identified based on two factors: **usage intensity** and **consistency**. Super-users are identified for each tool, based on the previous 28 days of activity. A super-user must meet both of the following criteria: 1. **Super-users must be in the top 10% of usage on that specific tool.** We look at user activity in each tool. We first look at the top 10% of users by cost of queries – because this metric is more consistent across tools and providers. If users are tied on cost, we break ties using input tokens (which reflect how much context was sent to the AI), followed by lines changed. Lines changed is the least preferred because some AI queries won't generate any lines changed, so it's not as consistent as cost and tokens. 2. **Consistent activity over the measurement period.** Users must have at least 10 days of active use on a tool to qualify as super-users. This dual-criteria approach ensures we identify users who are both heavily engaged and consistently using AI tools, rather than those with occasional spikes in usage. Someone could be a super-user for multiple tools too, in which case we would show them as a super-user for each of those tools. #### How to use information on super-users * Identify and interview your super-users to understand their workflows * Document specific prompts and practices that work on your codebase * Create peer-to-peer learning opportunities where super-users demonstrate real examples * Scale proven practices in playbooks and standards (e.g., in shared rules or markdown files that you recommend people put into LLM memory) Software engineers trust their peers more than external hype – and sometimes even more than they trust leaders. Showing how AI works on your actual codebase, shared by team members they respect, is one of the most effective things you can do to accelerate adoption. ## Run an AI Impact Survey
With the pace of change in AI, it’s extra important to understand how your team is feeling about these tools, and how they're using them. That's why we offer an AI survey in Multitudes. **What you can find out by running this survey** * How satisfied your people feel with AI * What use cases people are using AI for * The AI skills your team feels most and least effective with * How people perceive that AI is impacting their productivity and hours worked; we'll then help you cross-compare this with our telemetry data #### How to run an AI survey 1. **Make a copy of the survey:** Clicking this link will prompt you to copy our template [Google Form](https://docs.google.com/forms/d/17eKRLc7eouXHeHEQfAy2SdrIqCKNLedv2DEeYZziRnI/copy). Running the survey yourself ensures all data collected belongs to you. If you want the survey template in a different format besides Google Form, reach out to . 2. **Make edits to the form:** You'll need to edit the description; note that we've bolded areas you need to complete, for how you'll use the data and the due date. You are welcome to add additional questions – but note that we’ll only show the questions from the original template in the Multitudes app.
We recommend that you give people a couple weeks to complete the survey. 3. **Share the survey:** Share this survey with everyone at your org who writes code. You’ll likely need to remind people a couple times to get enough responses. **Getting results into Multitudes:** After you close the survey, share a spreadsheet of responses with # AI Impact Learn more about the impact your AI initiatives are having on your other Multitudes metrics. Understand how AI tools affect outcomes, by looking at their impact on leading and lagging indicators of productivity, quality, and developer experience. Multitudes conducts [ongoing research](https://www.multitudes.com/research) into the impact of AI on engineering teams. Our findings show that the actions you take as a leader have the biggest impact on the success of your AI rollout – not your tooling. With the right initiatives, you can help your team get more benefit from AI, with fewer of the costs (to codebase quality, learning, and more). But to do that, we need to be able to measure and compare the impact of each of our AI initiatives. This feature helps you do just that, with holistic metrics looking across productivity, code quality measures, and developer experience. Note that we found it is important to control for interventions by looking at pre-and-post metrics. We share more about that below. ## High and Low AI Usage Cohorts
Box and whisker chart comparing High AI users, low AI users and combined.
Multitudes automatically classifies users into two cohorts based on their AI tool usage patterns over the most recent 12 weeks: * **High AI Adopters**: Users with AI activity that is greater than or equal to (≥) the target percent over the days in the last 12 weeks * **Low AI Adopters**: Users with AI activity that is less than (<) the target percent over the days in the last 12 weeks #### Why is the default target 35%? Based on our recent [AI impact research](http://multitudes.com/data-to-cut-through-the-hype), we found that 50% Daily Active Users (DAU) was a strong predictor of meaningful AI usage when measuring only workdays (and excluding weekends). Since our feature calculates DAU across all calendar days (including weekends and holidays), we adjusted this threshold to 35% to reflect realistic usage patterns while maintaining the same signal of engaged adoption. (5 weekdays / 7 days of the week = 71% – so 50% of that is \~35%). A [custom target](/configuration-and-setup/customize-targets) can be set on the AI impact page if desired. #### Calculation Notes * Cohorts are defined **globally across your entire organization**, not per team. This means that you have a consistent view of who's a high AI adopter across teams. When you apply team filters, you'll see a subset of these global cohorts. * Historic team changes will not affect the cohorts due to them being filtered based on the current team state. This ensures we are comparing the same team members in each cohort in the pre and post intervention periods. * If multiple AI tools are integrated, DAU is calculated across **all tools combined**. * Low AI Adopters who consistently increase their usage will eventually graduate to the High AI Adopters cohort — this is a sign of adoption initiatives working. * Currently we are unable to provide an "exclude weekends" option as the data we obtain from most providers has already been aggregated at a daily level in UTC. The lack of granularity here means that we are unable to redistribute it to the actual days in local timezones. ## Adding AI Interventions ### Why this matters Success with AI isn't one-and-done; it's not that you roll out an AI tool and your work is complete. The pace of change with AI means that we need to treat this like a learning exercise – running experiments, and then iterating to help our teams get more out of these tools. That's why we allow you to add AI interventions in Multitudes. This might be the date you rolled out an AI tool, but it might also be an enablement activity, like the date you ran an AI hackathon or the date you started doing weekly AI demos on your team. Adding these interventions in Multitudes are the top thing you can do to help your team learn more about what works with AI – because when you add them, we calculate the impact of that specific intervention. Along the way, we bake in all the best practices from our [AI impact research](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/6941b0b2e9129ebfdfa71d4f_864f3a86793178cb6d8dcc1b464038be_What%20matters%20most%20for%20AI%20rollouts%20How%20you%20lead%20-%20Multitudes%20Whitepaper.pdf), including showing you a holistic set of outcome metrics, across productivity, quality, and developer experience. This helps you see what trade-offs might be happening. ### How to add an AI intervention There are multiple ways to add an AI intervention: #### (1) From the AI impact page The easiest way to add an AI intervention is from the [AI impact page](https://app.multitudes.co/ai-impact/impact). Go there, then click the "add intervention" button on the top right corner of the screen.
This will pop-up a modal for you to include the details of your intervention such as the intervention date, which team(s) this relates to, and a description. Ensure the "Is this an AI intervention?" box is checked before pressing "Add annotation".
Once you've done this, you can select the intervention from the AI intervention dropdown and then the charts on the page will update to show the impact of that intervention.
#### (2) From charts You can also add an AI intervention using the annotations feature available on most of our charts. Hover your mouse over the bottom axis of a chart, and you will see a prompt to add an annotation. Click "Add annotation", and the same modal as above will appear. Fill in the details of your intervention, ensuring that the "Is this an AI intervention?" box is checked.
### How the charts change with interventions **When no intervention is selected:** Charts will show metrics for Low AI Adopters, High AI Adopters and all AI users for comparison. **When an intervention is selected:** Charts will show Pre (12-week period leading up to the internvetion date) and Post (the full period after the intervention date) metrics for Low AI Adopters, High AI Adopters, and all AI users for comparison ## Measuring Impact on Adoption
This time series chart tracks Daily Active Users (DAU) over time, broken down by the "High" and "Low" AI adoption cohorts. It helps you understand adoption momentum and, when interventions are present, how they've impacted AI adoption. This is important to measure because we look at other impact metrics, because if an intervention didn't increase AI adoption, then we can't claim that it led to the follow-on outcome metrics. #### What You See The chart shows: * **Two trend lines**: One for High AI Adopters, one for Low AI Adopters * **Current DAU percentage** for each cohort * **Intervention markers** (when annotated) show when your organization introduced new tools, training, or process changes ## AI Impact Measurement
Box and whisker chart comparing Change Lead Time for High and Low AI users pre and post AI intervention.
Using box-and-whisker plots, we show how AI interventions impacted key performance metrics. The page supports two analysis modes: cohort-based comparisons (viewing High vs Low AI Adopters at a single point in time) and pre/post intervention analysis (tracking how each cohort changes after a specific action). Pre/post analysis provides stronger evidence for causality by controlling for pre-existing differences between groups, but we default to the simpler high vs low AI adopter view because the pre/post data isn't always available. #### Why We Use Pre/Post Intervention Analysis A common mistake in measuring AI impact is simply comparing High AI Adopters to Low AI Adopters and attributing all differences to AI usage. **This approach can be misleading.** The people who choose to use AI more are likely different from those who use it less — and these differences existed *before* AI was introduced. For example: * High AI Adopters might be more productivity-focused or excited about new technology * They might work in codebases or languages where AI performs better * They could be newer to a project and seeking help getting up to speed These pre-existing differences create [selection bias](https://catalogofbias.org/biases/selection-bias/). The high and low usage groups likely started with different metrics even before AI existed, which means comparing them directly confounds AI's actual impact with these other factors. For more about this issue, [read our blog post](https://www.multitudes.com/blog/practical-stats-how-to-actually-measure-ai-impact). {% hint style="info" %} **Real-world example**: In one organization we worked with, High AI Adopters initially appeared to have smaller PR sizes than Low AI Adopters — suggesting AI reduced PR size. But when we examined pre-intervention data, we discovered High AI Adopters *started out* with much smaller PRs before the AI rollout. Post-intervention, their PR sizes actually *increased* compared to their our pre-AI data. Without controlling for pre-existing differences, we would have made the wrong conclusion about the impact of AI. {% endhint %} This is why Multitudes emphasizes **pre/post intervention analysis**: by comparing each cohort to their own baseline, we control for pre-existing differences, which can help isolate AI's actual effect. #### When you don't have intervention data If you're viewing the page without configuring an intervention, you'll see direct comparisons between High and Low AI Adopters. This view is still valuable for understanding patterns and generating hypotheses, but remember: * **Be cautious about causality**: Differences could come from pre-existing factors, not AI itself * **Use it for exploration**: Identify interesting patterns worth investigating further * **Consider setting up an intervention**: Our AI impact research showed that what drives AI adoption isn't tool availability but enablement – so we recommend everyone run an AI intervention. And even small experiments (like a training session) create natural pre/post periods that strengthen your conclusions about Ai impact. The charts comparisons help you spot where differences exist, whereas running an intervention can help you understand *why* differences exist. #### How metrics are visualized Different metrics use different visualization approaches based on how the data is measured: **Box-and-Whisker Plots**: We use these when we have enough underlying event data to construct a box-and-whisker. We use this for charts where we have individual, event-level metrics – specifically: * PR Size: Each individual PR has a measurable size * Change Lead Time: Each individual PR has a lead time **Bar Charts**: We use these when the given metric is an aggregate, so it makes more sense to roll up all the data over the relevant time period (because we need a larger observation window before there's enough data for a meaningful box-and-whisker). We do this for charts including: * Merge Frequency: This aggregates based on PRs per week/month and controls for different-sized teams by dividing by contributor count. * Feedback Quality Given: This chart aggregates based on the quality of feedback (e.g., count of highly specific reviews or minimal reviews) divided by the total number of reviews. * Change Failure Rate: This chart aggregates by looking at the number of failures divided by the total number of changes. * Out-of-Hours Commits: This chart, like Merge Frequency, aggregates based on a count per week/month and divides by contributor count to control for different team sizes. ### Understanding Box-and-Whisker Plots Box-and-whisker plots visualize the **distribution** of data, helping you understand not just the average, but the full range of typical values. #### Reading the chart * **The box** represents the middle 50% of values (from the 25th to 75th percentile) – this range is called the interquartile range (= the 75th percentile value minues the 25th percentile value) * **The line inside the box** shows the median (the 50th percentile) — the mid-point value for that group. 50% of the datapoints sit above this line and 50% sit below it. * **The whiskers** extend to the largest data point within 1.5 times the interquartile range from the quartiles. Values beyond this are considered outliers. **The key thing to remember is that height matters**: Taller boxes indicate more variability in the data. Shorter boxes suggest more consistent behavior. ### Interpreting the chart insight percentage
Box and whisker chart showing PR Size, highlighting how percentages are shown based on changes pre and post intervention.
#### **Without intervention: Low vs high AI adopters** When no intervention is selected, the percentage shown in the insight compares High AI Adopters with Low AI Adopters. We calculate the percentage this way: $$ \text{Difference %} = \left(\frac{\text{median(High AI)}}{\text{median(Low AI)}} - 1\right) \times 100 $$ **Example**: If High AI Adopters have a median PR size of 150 LOC and Low AI Adopters have 200 LOC, the difference is (150/200 - 1) x 100 = -25, or -25%. This means that High AI Adopters create PRs that are 25% smaller than Low AI adopters. Note that while this difference is interesting, we cannot conclude it is because of AI usage as there were likely pre-existing differences between your low and high AI adopters (more in this blog post: [Don't measure AI impact by comparing low & high AI adopters](https://www.multitudes.com/blog/practical-stats-how-to-actually-measure-ai-impact)). #### **With intervention: Pre vs Post for** **Low and High AI adopters**
Image showing 4% increase for high AI adopters as 21 lines added per PR.
When an intervention is selected, we show three percentages: * the change for High AI Adopters * the change for Low AI Adopters * the change for all AI users Each percentage is calculated using the same relative-change formula as above, but instead of comparing High vs Low groups, we compare the same group before and after the intervention: When we have an intervention, we change the calculation to the following. $$ \text{Change%(High AI)} = \left(\frac{\text{median(High Post)}}{\text{median(High Pre)}} - 1\right) \times 100 $$ $$ \text{Change%(Low AI)} = \left(\frac{\text{median(Low Post)}}{\text{median(Low Pre)}} - 1\right) \times 100 $$ $$ \text{Change%(Combined)} = \left(\frac{\text{median(All Post)}}{\text{median(All Pre)}} - 1\right) \times 100 $$ By comparing each group to itself over time, this approach reduces the impact of pre-existing differences between developers or teams. That gives us more confidence that the changes we observe are associated with the intervention, rather than just reflecting differences that were already there. This is not as strong as a randomized controlled trial, which remains the best way to measure causal impact. But in practice, this pre-vs-post approach is a useful and more practical way to understand how outcomes change after an AI intervention. ### Industry benchmarks for AI Impact metrics Based on [our research](http://multitudes.com/data-to-cut-through-the-hype) analyzing engineering teams before and after AI adoption, we observed what the typical variation is across DORA, SPACE, and other metrics when a company does an AI intervention.\ \ **Merge Frequency** * Expected uplift for high AI adopters: 27.2% increase * Teams using AI tools more frequently showed significantly higher merge rates compared to low adopters. **Out-of-Hours Commits:** * Expected uplift for high AI adopters: 19.6% increase * Note: We're not saying that this is a good outcome, but that it's a typical outcome based on our research. If you notice an increase in out-of-hours commits, it's worth checking in with your team about what might be driving this – it could reflect different working patterns with AI tools, it could show that people are enjoying using AI tooling, or it could show that people are feeling the pressure to deliver more with AI. **Other Metrics (LOC, Lead Time, etc.):** * There was no consistent impact from AI interventions, at least not in our initial research. Instead, we saw wide variation in these metrics depending on the organization's specific practices. * We'll continue to monitor the changes from AI interventions and will update these benchmarks over time. **Important Caveats:** * These benchmarks are based on our AI impact study across multiple organizations. * Individual organization results may vary significantly – our research showed that each organization's AI journey is highly contextual. * We will continue to refine these benchmarks as we collect more data over time # Process Metrics Learn about common flow and velocity metrics, including all 4 key DORA metrics. Our process metrics cover most common flow and velocity metrics, including all 4 key DORA metrics. We also aim to make our metrics as actionable as possible - so we try to show possible causes of trends, like bottlenecks during reviews, large PR sizes, and whether the team's focus is aligned with the highest priority work. Complementing speed of delivery, we're also interested in quality - how often bugs are released, and how quickly systems are restored after a failure. ## Learn more about our Process Metrics
Flow of Work/pages/kefB0m7E6aeloFs4t1J4
Value Delivery/pages/ZuU3xuY5xxLJB2q5fC4Y
Quality of Work/pages/CF7b93IG2dSigxJDh4Aw
# Flow of Work Key metrics that track how quickly the team collaborates to deliver work, where delivery is flowing smoothly, and where it’s getting blocked. ## Explore our Flow of Work metrics
Change Lead Time/pages/szAl5TjmIcKlQDq9xvcY
Coding Time/pages/inOZIGknRH8H1eaIBcjP
Review Wait Time/pages/fB4JK9Yc7SwWOe2ohSJL
Editing Time/pages/Tdt6AZN6MypssVVhue94
Deploy Time/pages/3EJkhYzXFhm29f0u5yxY
PR Size/pages/xWejiriMw0mZFRwHeqhE
Focus Time/pages/AbEUnaoytopCzoxFsKds
### Filtering Flow of Work metrics All Multitudes metrics can be filtered by date range, team/s and repositories. You can also turn on the **Exclude weekends** filter to exclude non-working days from calculations. This respects each person's individual schedule – so if someone works part-time Monday to Wednesday, their "weekend" would be Thursday through Sunday. You can set up [custom working days](/configuration-and-setup/configuring-working-hours) for each team member in [Settings > Team Members](https://app.multitudes.co/team-settings). You can switch between daily, weekly, or monthly views to match your reporting needs. You can also switch between P50, P75, P90, or P95 percentiles to see the spread of your data. # Change Lead Time {% hint style="info" %} ⭐️ This metric is one of the [4 Key Metrics published by Google's DevOps Research and Assessment (DORA)](https://dora.dev/guides/dora-metrics-four-keys/) team. You can see all 4 key DORA metrics on the [DORA Metrics page](https://app.multitudes.co/DORA) of the Multitudes app. {% endhint %} ![Change Lead Time graph](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/6739fb3d6f5f42c0839a9dbc_flow-of-work-leadtime.png) ***‍*****What it is:** This is DORA's `Change Lead Time` (previously called `Lead Time` in the app), a metric that shows how long it takes the team to write code, request and give feedback, make revisions, merge the PR, and then deploy into production. It’s an indicator of how long it takes to deliver value to customers.**‍** **Why it matters:** `Change Lead Time` is one of the top four indicators of software team performance, according to [Google’s DORA research](https://cloud.google.com/devops/) (the DevOps Research and Assessment research). Their research shows that a faster `Change Lead Time` is correlated with better business outcomes. Specifically, teams with a faster `Change Lead Time` do work that is better, more stable, and more secure. If you want to dive deeper into this, check out the [Accelerate](https://itrevolution.com/book/accelerate/) book. Note that this is closely related to `Cycle Time`; we measure `Change Lead Time` since that's recommended by DORA. [See here](https://www.multitudes.com/blog/lead-time-vs-cycle-time) for more about the differences between `Change Lead Time`, `Cycle Time`, and `Lead Time`. **How we calculate it:** To calculate `Change Lead Time`, we measure the **number of hours from the first new commit** on a pull request’s (PR’s) branch **to a successful attempt of deployment to production for that PR, or to PR merged** if there is no deployment data.‍ {% hint style="info" %} This means some of the PRs included in this chart will not yet be deployed. This is so that you can get insights for all repositories, even ones that don’t have workflows configured. You can get a breakdown of how long PRs are taking to merge vs. deploy with the Change Lead Time subsets chart or the specific line charts (like [this one for deploy time)](/metrics-and-definitions/process-metrics/flow-of-work#deploy-time). {% endhint %} {% hint style="info" %} Because the end point for Change Lead Time is a merge (or a successful attempt to deploy to production if available), the metrics will not include PRs that were closed (i.e not merged) {% endhint %} {% hint style="success" %} **What good looks like** The latest DORA research ([from 2025](https://dora.dev/research/2025/)) shows that the top 15% of responses have a `Change Lead Time` of less than 24 hours. {% endhint %} ### **Additional calculation notes for Change Lead Time & subsets** Here are a few additional notes that affect the calculations for `Change Lead Time` and/or its components, `Coding Time`, `Review Wait Time`, `Editing Time` and `Deploy Time`. These exclusions apply to Coding Time, Review Wait Time, and Editing Time : * Our focus is on PRs that the team collaborated on, so we exclude bot merges and selfie-merges (PRs merged by the PR author, with no comments or reviews by other collaborators). * If you like, you can choose to exclude weekend hours from these calculation; simply toggle on “Exclude Weekend Hours”. For just `Change Lead Time` and `Coding Time`, please note: * When you first join Multitudes, your historical data (the first 6 weeks) will show a lower time metric if your company does a lot of rebasing. This is because we can’t get original commits from the historical data in GitHub, so the rebased commit is taken as the first commit. * Once you integrate, we get events data from GitHub. This means we will get the original commits that are pushed to GitHub, even if your teams rebase or squash the commits later. Therefore, you might notice that your metrics are higher after the time that you onboard onto Multitudes, compared to your historical data. * If you rebase your commits before pushing the branch to GitHub for the first time, GitHub notifies us of the rebase time rather than when the commits were originally made. In order to provide the most accurate measure of `Coding Time`, we therefore encourage you to push commits to GitHub as they are made, even if the intent is to rebase or squash these commits before opening a PR or requesting a review. For `Review Wait Time`, please note: * The counting for the `Review Wait Time` starts when a review is first requested. If a review was never requested (i.e. when someone provides a review without being asked to review), we fall back to counting from the time the PR was first opened or when it was first moved to "Ready for review" if it was initially a draft PR. For just the separate `Deploy Time` line chart: * If you’re using the [GitHub Actions integration](/integrations/github-actions)… * We do include selfie PRs (non-collaborative PRs) in this metric. * This is because we want to provide the most comprehensive view of how long your deployment pipeline takes, which means incorporating all relevant data. * Whether a deploy is of a collaborative PR or a selfie PR, this should not normally influence how long the deploy takes, therefore selfie PRs are a relevant segment of your team’s data on how long things take to deploy. * If you're using the [Deployments API](/integrations/deployments-api), we only look at what you POST, so you can decide how you want to treat selfie PRs * For `Change Lead Time` we intentionally include all merged PRs, even if they are not yet deployed (or if [deployment metrics aren’t available](/metrics-and-definitions/deployment-metrics)), so that you are getting insights across all repositories, even if some of them are not set up with GitHub Actions deploy pipelines or the Deployments API. ### **How Change Lead Time is broken into its subsets** `Change Lead Time` is broken up into its four components, `Coding Time`, `Review Wait Time`, `Editing Time`, and `Deploy Time`. Where these start and end can depend on the events in each PR's life cycle. Here is a typical PR timeline. Click the dropdown below it for more scenarios.
Timeline showing Change Lead Time begins at the First new commit and ends when deployed.

Timeline showing Change Lead Time begins at the First new commit and ends when deployed.

{% hint style="info" %} The Change Lead Time by subset chart in the app shows the percentile value for each component, `Coding Time`, `Review Wait Time`, `Editing Time`, and `Deploy Time`. So if it’s showing P50, the 50th percentile or median, the chart shows that for each sub-component.\ \ The sum of these components may not equal the overall `Change Lead Time` shown in the Change Lead Time line chart because of how percentiles work. {% endhint %} ## More PR life cycles scenarios {% tabs %} {% tab title="Scenario 1" %}
{% endtab %} {% tab title="Scenario 2" %}
{% endtab %} {% tab title="Scenario 3" %}
{% endtab %} {% tab title="Scenario 4" %}
{% endtab %} {% endtabs %} # Coding Time {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Coding time graph](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/640694a7da6593f88637c07b_Coding%20Time.jpg) **What it is:** This shows how long the team spends writing code before asking for feedback. **Why it matters:** `Coding Time` represents the first part of the development cycle in a team, and it can be a bottleneck that increases `Change Lead Time`. There are many reasons why `Coding Time` might be high, e.g. poorly-scoped work, external interruptions leading to less focus time, or a complicated piece of work. **How we calculate it:** This measures the number of hours from the **first new commit on a PR** to when the **PR is ready for review**. * What defines the first *new* commit on a PR? * We exclude commits that were created earlier on other branches, and then pulled in to the PR’s head branch. * We take into account all commits that were pushed to GitHub, even if they are later squashed in a rebase. This means that even if you squash all your commits on a PR down to 1 commit before merging, we will still use the timestamp of your first *original* commit as the start of coding time, as long as the original commit was pushed previously. * If you rebase your commits before pushing the branch to GitHub for the first time, GitHub notifies us of the rebase time rather than when the commits were originally made. In order to provide the most accurate measure of `Coding Time`, we therefore encourage you to push commits to GitHub as they are made, even if the intent is to rebase or squash these commits before opening a PR or requesting a review. * What happens if first new commit time is after PR creation time? * If the first *new* commit on a PR comes after PR creation time, then the PR creation time is taken as the start of coding time, rather than the time of first commit. * This is so that `Coding Time` can capture the entire “draft time”. It makes sense to include the time that the PR spend in "draft" in this measure of time spent coding. * If the PR was created in a non-draft state, `Coding Time` is `null`. This is because it means the PR was ready for review upon creation, and `Review Wait Time` (the next metric in the PR life cycle) starts at the point where the PR is first ready for review. * See [here](/metrics-and-definitions/process-metrics/flow-of-work/change-lead-time#additional-calculation-notes-for-change-lead-time-and-subsets) for additional notes on how this metric is calculated, from `Change Lead Time`. {% hint style="success" %} **What good looks like:** We recommend that `Coding Time` be under **4 hours**. This threshold is based on an internal analysis conducted by Multitudes across 80,000 PRs from a diverse range of customers and comparing against the SPACE and DORA research. {% endhint %} # Review Wait Time ![Review Wait Time graph](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/634800eb3c45b5629e3e4219_Review%20Wait%20Time.jpg) **What it is:** This shows how long people wait to get feedback on their PRs. **Why it matters:** This is one possible bottleneck for `Change Lead Time`. When people have to wait longer for feedback, it can mess up their workflow. They’re more likely to start a new piece of work while waiting for feedback. When they get that feedback, they have to context-switch, making it harder for them to remember what they did. This often results in longer times taken for each of the tasks to be completed (for example, [one study showed](http://blog.ninlabs.com/2013/01/programmer-interrupted/) that it takes 10-15 minutes to get back into context). Moreover, there’s bias in how long different groups of people have to wait for feedback. For example, [this research](http://www.erinhengel.com/research/publishing_female.pdf) showed that women had to wait longer than men for feedback. This is why we *do* show this metric at the individual level — so that you can make sure that everyone is receiving feedback in a timely manner. **How we calculate it:** We measure the **number of hours from PR creation until the PR gets feedback**. This could be a comment, review, or merge by someone other than the PR author. It excludes time that the PR spends in a draft state, since the draft state indicates that the PR author is still finishing the work. To be clear on some nuances: * `Review Wait Time` is `null` if the PR has no feedback. It ignores responses from bots and responses that came in after the merge (since we exclude selfie merges). * See [here](/metrics-and-definitions/process-metrics/flow-of-work/change-lead-time#additional-calculation-notes-for-change-lead-time-and-subsets) for some additional calculation notes that apply from `Change Lead Time`. {% hint style="success" %} **What good looks like** We recommend that `Review Wait Time` be under **4 hours**. This threshold is based on an internal analysis conducted by Multitudes across 80,000 PRs from a diverse range of customers and comparing against the SPACE and DORA research. {% endhint %} # Editing Time {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Editing Time graph](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/640697ca8fabd66171401973_Editing%20Time.jpg) **What it is:** This metric shows how long code takes to get merged once feedback has been received. **Why it matters:** As a measure of back-and-forth between the code author and those who are reviewing the code, `Editing Time` is important for understanding bottle-necks in `Change Lead Time`. A high `Editing Time` could mean that the team needs to improve how they scope work, the received feedback is confusing, the PRs being created are large, or there are other distractions preventing fast iteration. A low `Editing Time` indicates that the team is able to quickly action feedback and ship work once it has been reviewed. **How we calculate it:** We measure the number of hours from first feedback on the PR to PR merge, i.e. the back-and-forth editing time. If there was no response before the merge, `Editing Time` is `null`. See [here](/metrics-and-definitions/process-metrics/flow-of-work/change-lead-time#additional-calculation-notes-for-change-lead-time-and-subsets) for some additional calculation notes that apply from `Change Lead Time`. {% hint style="success" %} **What good looks like** We recommend that `Editing Time` be under **16 hours**. This threshold is based on an internal analysis conducted by Multitudes across 80,000 PRs from a diverse range of customers and comparing against the SPACE and DORA research. {% endhint %} # Deploy Time {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} Deploy Time line graph ***‍*****What it is:** This metric shows how long code takes to get deployed once a PR has been merged. **Why it matters:** `Deploy Time` measures how fast new features, bug fixes, and hot-fixes can reach end-users. A lengthy build or deployment process can delay crucial updates, hinder customer satisfaction, and even impact the overall competitiveness of a product or service. By minimizing `Deploy Time`, organizations can ensure a faster time-to-market. A long `Deploy Time` also has a direct impact on developer productivity and well-being. As this can introduce frustration and inefficiency, leading to a decline in developer morale and motivation. When developers have to wait for long periods to see their code build, it can disrupt their workflow and hinder their ability to iterate and make further improvements. **How we calculate it:** The number of hours from when a PR is merged to when an [attempt to deploy it to production](/metrics-and-definitions/deployment-metrics) succeeds. **A few notes:** * If you have GitHub Actions set up, see [here](/integrations/github-actions) for info on how the way you use and configure this integration will appear in your data * If you’re using our [Deployments API](/integrations/deployments-api), the commitSha is what we’ll use to match deployments to PRs # PR Size {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![A stylized line graph shows time to merge is high at 18 hours but is trending up.](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/634800ebfff45604484ae35d_PR%20Size.jpg) ***‍*****What it is:** How large your team's PRs are. We show two representations of this — `Lines Changed` and `Files Changed`. **Why it matters:** This is another possible bottleneck for `Change Lead Time`. We know that large PRs are harder to review, test, and manage in general. It is now generally accepted that keeping PR size down is best practice for faster reviews, less merge conflicts (and therefore easier collaboration), and simpler rollbacks if required. Learn more in [this 2017 paper by Microsoft and the University of Victoria](http://chisel.cs.uvic.ca/pubs/macleod-IEEESoftware2017.pdf), and in [Google’s own internal guidelines](https://google.github.io/eng-practices/review/developer/small-cls.html) (they say “changelist” rather than “pull request”).**‍** **How we calculate it:** We show the **selected percentile (P50, P75, P90, P95) of the lines of code or the number of files changed per PR** depending on the option selected. We chose to provide 2 options here (instead of just lines of code) so you can get a more well-rounded view of the overall size. We recognise that these are both simple measures of "PR Size" which don't take into account edge cases such as lock files or automated formatters (examples where PR size may be large, but the PR is still easy to review and manage). However, in the majority of cases, the number of lines or files changed is a reasonable indicator of how long a PR may take to get merged. {% hint style="success" %} **What good looks like** Many organizations like to enforce maximum limits on the lines of code (LOC) changed per PR, generally ranging from around 200 to 400. [This study](https://smartbear.com/learn/code-review/best-practices-for-peer-code-review/) also found that PRs should be limited to 200-400 LOC; beyond that, the ability to effectively capture defects goes down. So we recommend keeping LOC under 300 as a good middle ground. {% endhint %} {% hint style="warning" %} Note: `Files Changed` varies - you can have a small number of LOC changed across many files, and it'd still be fairly easy to review. In our teams, we try to keep it under 10 files changed. {% endhint %} # Focus Time
***‍*****What it is:** This shows how people’s working hours were distributed across meeting time, fragmented time, and focus time. Meeting time is what it sounds like – the hours we spend in meetings. [Focus time](https://www.getclockwise.com/blog/what-is-focus-time) is a 2+ hour block of uninterrupted working time during Multitudes working hours. Fragmented time is a period without meetings but less than 2 hours long.\ \ You will need to [integrate with Google Workspace](/integrations/google-calendar) or [Outlook](/integrations/outlook-calendar#outlook-calendar--multitudes-integration-benefits) to get this metric. **Why it matters:** [Extensive research](https://hbr.org/2024/06/hybrid-work-has-changed-meetings-forever) shows that changing how we work to include more deep work time can increase productivity. In addition, it’s not just about time spent in meetings, but how they’re spread across the day that affects people’s ability to do deep work.**‍** **How we calculate it:** First, we look at someone’s working hours. By default, this is set to 8am-6pm weekdays, local time. To support flexible hours, our metric is configurable for different time zones and different preferred working hours and days on each team member's profile in [Settings](https://app.multitudes.co/team-settings). (Note that any changes will only impact future time calculations.) {% hint style="warning" %} Note: To maintain accuracy, your calendar data is synced with a 48-hour delay. This helps us capture any changes made to past events. {% endhint %} Within someone’s working hours, we split their time into meetings, focus time, and fragmented time: * **Meeting time:** We count an event as a meeting if 2+ people RSVP’d yes. This means that if you use your calendar for time blocking or personal reminders, that won’t affect our calculations. If meetings overlap, we take the duration of the time the person was occupied rather than the duration of both meetings added together. For example, if you had a meeting from 10am - 11am and another from 10:30am - 11:30 am, we would calculate the meeting hours as 1.5 hours rather than 2 hours. **We exclude the following events from our categorisation of meetings:** * Events with a single attendee * Events that are over 12 hours - as these are usually placeholders for things like strategy days * Events marked as "free" or "maybe"/"tentative" - even if multiple people have accepted the event invite * Non-meeting event types, such as “focus time” * Events with keywords and phrases in the title that indicates it is a non-meeting event. These patterns include: deep work, focus time, no meetings, meeting-free time, blocked time, block time, blocked, please do not schedule, do not schedule, busy * **Focus time:** This is 2+ hours of time between the end of a meeting and the start of the next one. * **Fragmented time:** This is any block of less than 2 hours without meetings. For example, 15 mins in between meetings would be fragmented time. The chart is displayed in the organization timezone so if you have team members spread across time zones, you may see slight differences when looking at the Day aggregations. {% hint style="info" %} **How we handle Out of Office events:** * If you are Out of Office during working hours, your meeting time, fragmented time, and focus time will be calculated based on the remainder of working hours. * To view Out of Office hours click the chart to “view details”. If a team member was out of office, the drilldown will show the number of hours they were out for. * If an Out of Office event overlaps with a meeting, the meeting will take precedence. * Events with the following keywords and phrases in the title are classified as Out of Office hours: lunch, lunch break, break, coffee break, travel time, commute {% endhint %} {% hint style="info" %} **How we handle Group Meetings in Outlook:** * If you and 1+ other person are both marked “Busy”, then it’s a meeting. * If the event is Free/ Working Remotely / Tentative / Out of office, then we don’t count it as a meeting. {% endhint %} {% hint style="success" %} **What good looks like** In our benchmarks, we see 85% focus time for the 75th percentile of people – this means that the top 25% of people get at least 85% focus time. For an 8-hour workday, that's 6.8 hours of focus time, and 1.2 hours of meeting or fragmented time. Note that this benchmark is primarily influenced by the patterns for individual contributors. It's great that this is so high – it means that these people are getting the 4+ hours of "Maker Time" that productivity expert [Cal Newport](https://www.getclockwise.com/blog/what-is-focus-time) and Y Combinator founder [Paul Graham](https://www.paulgraham.com/makersschedule.html) both recommend. If you're not an individual contributor, we recommend 2+ hours a day of `Focus Time` on average. This is based on [HBR research](https://hbr.org/2013/09/make-time-for-the-work-that-matters) suggesting at least 10 hours/week (2 hours/day) of focus time. [Clockwise research](https://www.getclockwise.com/blog/what-is-focus-time) also showed that most engineering managers want at least 1-3 hours of focus time per day; 2 hours is also the midpoint of that. For everyone, we recommend that fragmented time should be as close to 0 as possible since it’s hard to do useful work in these short periods. Multiple weeks of low focus time or an decrease in the amount of focus time may indicate your team isn't getting enough time to work and feel productive. {% endhint %} # Value Delivery Key metrics that track your team's output through deployment rates, merge frequency, and work type distribution. ## Explore our Value Delivery metrics
Deployment Frequency/pages/kTHLtxDI6qXxqmU86Crv
Merge Frequency/pages/FSiJ7Bmiw7hs6detNHFl
Types of Work/pages/x8PMw2qhkqQheblrYE7a
Feature vs Maintenance Work/pages/RJXQQ3iTqfBc0REXzOXD
### Filtering Value Delivery metrics All Multitudes metrics can be filtered by date range, team/s and repositories, and switched between daily, weekly, or monthly views to match your reporting needs. # Deployment Frequency {% hint style="info" %} ⭐️ This metric is one of the [4 Key Metrics published by Google's DevOps Research and Assessment (DORA)](https://dora.dev/guides/dora-metrics-four-keys/) team. You can see all 4 key DORA metrics on the [DORA Metrics page](https://app.multitudes.co/DORA) of the Multitudes app. {% endhint %} {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Deployment Frequency chart](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/6493d20986f5aee5f1ff5741_Deplyoment%20Frequency.png) **What it is:** This is [DORA's Deployment Frequency](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance), showing the median number of [successful attempts to deploy to production](/metrics-and-definitions/deployment-metrics) per person on a team, over time. *‍* **Why it matters:** This is an indicator of the value we're providing to customers, because it shows the volume of work being released to production. **How we calculate it:** We count the **number of successful attempts to deploy to production** in each time period, **divided by the number of people on the team**. This normalization is to allow benchmarking of teams against industry standards, regardless of team size. {% hint style="warning" %} **Notes:** If you send us a deployment with only bot commits, and those bot commits were last updated before you installed our GitHub integration, those custom deployments will not show up in the data. This is because we don't retrieve bot commits during the initial historical data pull when you first onboard. Commits that are authored by users who are not a [contributor](https://www.multitudes.co/help/login-access-permissions#data-inclusion-permissions) in Multitudes will be filtered out of results. If a deployment has no commits authored by Multitudes contributors, the whole deployment will be filtered out of results as we won't be able to attribute it to a Multitudes contributor nor their team. If a deployment only has commits authored by bots, these will be included in the grey Organization line only. {% endhint %} {% hint style="success" %} **What good looks like** In Google's DORA research, the top 15% have a `Deployment Frequency` that is on demand, with multiple deploys per day [(see DORA 2025](https://dora.dev/research/2025/)). If we call that one deploy per day per team, that’s 5 deploys per week in a 5-day workweek. We recommend keeping this metric over 2 deployments per person per week. {% endhint %} # Merge Frequency {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Merge Frequency graph](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/63480105db9cad1bbd9a2f5a_Merge%20Frequency.jpg) **What it is:** This is an alternative to [DORA's Deployment Frequency](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance). It shows the median number of PRs merged per person on a team, over time. **Why it matters:** This is an indicator of the value we're providing to customers, because it shows the volume of work being released to production. It can be useful to measure both `Merge Frequency` and `Deployment Frequency`. While it is best practice for each merge to automatically deploy to prod, often this is not the case - maybe your deploy pipeline is not yet fully automated, or it takes a while to deploy so you often want to batch changes. A significant difference between these two measures might indicate opportunities for improvement in your deployment processes. **How we calculate it:** We count the **number of PRs merged** in each time period, **divided by the number of people on the team**. This normalization is to allow benchmarking of teams against industry standards, regardless of team size. Merged PRs are attributed to the person who **opened** the PR, not the person who performed the merge action. We use "merged" to mean "completed," reasoning that the person who opened the PR is most likely to have done the work. You can filter for only collaborative PRs (ones that had input from someone other than the PR author) using the [Collaborative PRs toggle](/knowledge-base/collaborative-prs-and-all-prs-toggles). {% hint style="success" %} **What good looks like** [Google suggests that elite teams should be deploying multiple times a day](https://storage.googleapis.com/gweb-cloudblog-publish/images/Calculating_the_metrics_frOhcbp.max-2800x2800.jpg). If we call that one deployment per day per team, that’s 5 deploys per week in a 5-day work week. Dividing this by a rough approximation of team size (around 5 developers), and taking into account the fact that there's sometimes more than one PR included in a single deploy (for major features, it could be best practice to collect up lots of changes into a release branch), we recommend keeping this metric over 2 PRs merged per person per week. {% endhint %} # Types of Work {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} Bar chart showing types of work **What it is:** This shows how much of each type of work the team completed. To understand more about how “work” and “completed” are defined and customized, [click here](/configuration-and-setup/customize-work-categories). You can hover over a specific section of the bar chart to get more details on how many tickets were dedicated to that task. **Why it matters:** This metric gives you visibility over team velocity and how the team’s work was spread across different types of work. If a team is struggling to get their planned feature work done, this is a useful chart to consult to see what could be getting in the way, and understanding if the types of issues completed align with what was planned. {% hint style="info" %} When people are interrupted on a project, it can take up to [23 minutes](https://link.springer.com/book/10.1007/978-3-031-02212-8) to get back on track (e.g., fully shift their thinking, remember where they left off, etc.). The more projects an individual holds, the more they therefore have to “context switch”, which can [reduce overall productivity](https://insights.sei.cmu.edu/blog/addressing-the-detrimental-effects-of-context-switching-with-devops/), while also increasing feelings of [stress and frustration](https://www.ics.uci.edu/~gmark/chi08-mark.pdf). Across the team, the cost can really add up; one academic study found that developers working on 2+ projects spend 17% of their development effort on [managing interruptions](https://www.researchgate.net/publication/317989659_Impact_of_task_switching_and_work_interruptions_on_software_development_processes/link/5f4023b9a6fdcccc43e3a8d1/download). Did the team have enough time for feature work or did bug work get in the way? In one survey of \~1000 developers, 44% said [bugs were a key pain point](https://content.rollbar.com/hubfs/State-of-Software-Code-Report.pdf) in their day-to-day work and a main reason deployments were slow. {% endhint %} **How we calculate it:** this depends on your configuration, please [click here](/configuration-and-setup/customize-work-categories). {% hint style="warning" %} Note: The chart displays the top 5 projects by issue count. Projects beyond the top 5 are aggregated and shown as "Other projects" to maintain clarity and focus on the most significant work areas. {% endhint %} {% hint style="success" %} **What good looks like** This depends on your team and product priorities. Many teams value consistency week-to-week, since it helps with their planning. It can also be helpful to watch for increases in bug work, since that can decrease the team’s time for feature work. Overall, the goal of this chart is to make sure your team is working on the most important thing(s) and getting work done at a reasonable pace. {% endhint %} # Feature vs Maintenance Work Issue-tracking {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Bar chart showing feature vs maintenance work](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/6670cedb1f81010b735dcc6e_Feature%20vs%20Maintenance%20Work.jpg) **What it is:** This shows the relative percentage of either `Feature` or `Maintenance work` completed. For more information, see **How we calculate it** below. **Why it matters:** Delivery is about managing the balance between shipping new features and maintaining existing systems. If you neglect maintenance, your codebase and systems can [“rot”](https://en.wikipedia.org/wiki/Software_rot), slowing down delivery of new features and site reliability. On the other hand, spending too much time on maintenance can cause the team to miss delivery targets. This is why visibility over where your team is spending their time is important, to make sure that the balance reflects your priorities. **How we calculate it:** We may show multiple Feature vs Maintenance charts, depending on whether you have issue tracking integration(s) installed: * **For our Jira or Linear integration:** this depends on your configuration. [Click here](/configuration-and-setup/customize-work-categories) to understand more about how we define what is considered “work” and when it is considered "complete". * **For our GitHub integration:** this is based on all commits with [conventional commit](https://www.conventionalcommits.org/en/v1.0.0/) prefixes. * This includes all commits, not just the commits that are merged into production (which may be squashed commits). * You can filter for just conventional commits and/or just the commits on each repository's [default branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/changing-the-default-branch) with the toggles above the chart. * For the moment, we have set the default mapping using the most common [conventional commit](https://www.conventionalcommits.org/en/v1.0.0/) prefixes to our categories as follows: * Feature: `feat`, `perf` * Investment: `build`, `ci`, `chore`, `docs`, `refactor`, `style`, `test` * Bug: `fix` * Unassigned: all other commits e.g. if they use a different prefix, or don’t follow the conventional commits format. * You can customize the categories. Click the "Edit Categories" link under the legend, and either choose an existing category to "Edit", or "Add category". * You may select the prefixes counted towards a particular category using the dropdown of common conventional commit prefixes. Prefixes that are already used in another category will be disabled. * You can also type your own custom prefix strings. Press enter after typing each one. * There is also a checkbox labeled "Include GitHub revert commits in this category". This is for catching the commits created via the "revert" button on the GitHub UI, which creates a PR (and therefore an eventual commit if the PR is squash merged) in the format Revert "feat: old PR title with conventional prefix". {% hint style="success" %} **What good looks like** Many teams set aside an upfront “tech debt budget” or “maintenance budget” when planning upcoming work. Many will allocate 10-20% for maintenance, but this depends on the team. For example, teams focused on maintaining legacy code might budget 50% of work (whether story points or issues or commits) to maintenance. Another approach is to allocate specific days, such as 1 day every week or fortnight. To learn more check out this article on [how to define and spend your tech debt budget](https://hackernoon.com/how-to-define-and-spend-your-tech-debt-budget-8429z32h2) and this one on [reclaiming tech equity](https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/tech-debt-reclaiming-tech-equity). Once you have defined a budget, it’s easy to use this chart to track real world “spend”! {% endhint %} # Quality of Work Key metrics that track software quality and incident response, including failure rates, recovery times, and system stability measures. ## Explore our Quality of Work metrics
Change Failure Rate/pages/wIuLTjIduc8buNaug1sL
Mean Time to Recovery/pages/i8CLey0hIbtE0PMJzxig
Mean Time to Acknowledge/pages/9QQ2glQM99eT06ZTHAlm
Number of Pages/pages/b8zoE1ILGViHePPUepEu
Deployment Failure Rate/pages/E8ALZwJuFEvGexNS7fFY
### Filtering Quality of Work metrics All Multitudes metrics can be filtered by date range, team/s and repositories, and switched between daily, weekly, or monthly views to match your reporting needs. **If you use our PagerDuty integration, you can also filter your Incident metrics by either Priority or Urgency.** ![](/files/G2h3ojKOLNuZ8viDMFq1) **How it works:** * When [installing your PagerDuty integration](/integrations/pagerduty), your Multitudes owner will have selected your default filter setting based on whether it's most useful for you to filter incidents on urgency or priority level. * You can adjust this in your session by clicking the filter icon in the filter to switch between Urgency and Priority * You can then filter based on those levels This will impact the following charts: * [Mean Time to Recovery (MTTR)](/metrics-and-definitions/process-metrics/quality-of-work/mean-time-to-recovery) * [Mean Time to Acknowledge (MTTA)](/metrics-and-definitions/process-metrics/quality-of-work/mean-time-to-acknowledge) * [Number of Pages](/metrics-and-definitions/process-metrics/quality-of-work/number-of-pages) # Change Failure Rate {% hint style="info" %} ⭐️ This metric is one of the [4 Key Metrics published by Google's DevOps Research and Assessment (DORA)](https://dora.dev/guides/dora-metrics-four-keys/) team. You can see all 4 key DORA metrics on the [DORA Metrics page](https://app.multitudes.co/DORA) of the Multitudes app. {% endhint %} {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Change Failure Rate graph](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/638c21ca0787c76083dcc4e2_Change%20Failure%20Rate.png) **What it is:** The percentage of PRs merged that indicate that some kind of failure was released and had to be fixed. **Why it matters:** This is our take on [DORA's Change Failure Rate](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance), which indicates the percentage of deployments that cause a failure in production. It's a lagging indicator of the quality of software that is being deployed - how often does it contain bugs that cause failures later? **How we calculate it:** We calculate the % of merged PRs with specific keywords in the title. Specifically, we look at: (1) the number of merged PRs that contain any of the words `rollback`, `hotfix`, `revert`, or `[cfr]` in the PR title and then divide by (2) the total count of merged PRs (including selfie PRs, which are PRs merged by author with no comments or reviews from others). We tested and chose the keywords above to catch most cases where a PR was required to fix a previously-released bug, while minimizing false positives. When there was disagreement about whether to include a keyword (e.g., `fix`), we excluded it – so for most teams, our Change Failure Rate is a low estimate.
Tip: Use a script to automatically add [cfr] to PRs that meet certain criteria In the future we will allow customization of which key words we take into account when determining what constitutes a change failure. In the interim if your team has established practises around naming conventions, you can use a CI script to add `[cfr]` to the title of PRs you wish to track in Multitudes. Example 1: Github Actions job that checks if the PR **title** contains the word `fix`, and if so appends `[cfr]` to the title. {% code expandable="true" %} ```yaml name: Multitudes - Tag Fix PRs for Change Failure Rate on: pull_request: types: [opened, edited] # runs when PR is created or the title is changed permissions: pull-requests: write issues: write jobs: update-title: runs-on: ubuntu-latest steps: - name: Adjust title if necessary uses: actions/github-script@v7 with: script: | const pr = context.payload.pull_request; const originalTitle = pr.title; // Rules const hasFixPrefix = /^fix\(/.test(originalTitle); const hasCfrSuffix = /\s*\[cfr\]$/.test(originalTitle); let newTitle = null; let comment = null; if (hasFixPrefix && !hasCfrSuffix) { newTitle = `${originalTitle} [cfr]`; comment = "🤖 **Automated update**: Added `[cfr]` tag to PR title since this is a fix commit. This will help track 'change failure rate' in Multitudes."; } else if (!hasFixPrefix && hasCfrSuffix) { newTitle = originalTitle.replace(/\s*\[cfr\]$/, ''); comment = "🤖 **Automated update**: Removed `[cfr]` tag from PR title since this is not a fix commit. The `[cfr]` tag should only be used for fix commits."; } if (newTitle === null) { core.info("No update needed."); } if (newTitle !== null) { await github.rest.pulls.update({ owner: context.repo.owner, repo: context.repo.repo, pull_number: pr.number, title: newTitle }); await github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, issue_number: pr.number, body: comment }); core.info(`Updated PR title: "${originalTitle}" -> "${newTitle}"`); } ``` {% endcode %} Example 2: Github Actions job that checks if the PR **branch** contains the word `hotfix`, and if so appends `[cfr]` to the title. {% code expandable="true" %} ```yaml name: Multitudes - Tag Hotfix PRs for Change Failure Rate on: pull_request: types: [opened, edited] permissions: pull-requests: write issues: write jobs: update-title: runs-on: ubuntu-latest steps: - name: Adjust title if necessary uses: actions/github-script@v7 env: BRANCH_PREFIX: "hotfix" with: script: | const pr = context.payload.pull_request; const headBranch = pr.head.ref; const originalTitle = pr.title; const branchPrefix = process.env.BRANCH_PREFIX; // Rules const hasMatchingBranch = new RegExp(`^${branchPrefix}/`).test(headBranch); const hasCfrSuffix = /\s*\[cfr\]$/.test(originalTitle); let newTitle = null; let comment = null; if (hasMatchingBranch && !hasCfrSuffix) { newTitle = `${originalTitle} [cfr]`; comment = `🤖 **Automated update**: Added \`[cfr]\` tag to PR title since this is a ${branchPrefix} branch. This will help track 'change failure rate' in Multitudes.`; } if (newTitle === null) { core.info("No update needed."); } if (newTitle !== null) { await github.rest.pulls.update({ owner: context.repo.owner, repo: context.repo.repo, pull_number: pr.number, title: newTitle }); await github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, issue_number: pr.number, body: comment }); core.info(`Updated PR title: "${originalTitle}" -> "${newTitle}"`); } ``` {% endcode %}
{% hint style="info" %} We recognize that this proxies failures after the fact; this is because it's not actually possible to know if someone's releasing a failure into production in the moment, otherwise it wouldn't have been released in the first place. You can include the square-bracketed keyword `[cfr]` in your PR titles if you'd like more granular control over what gets counted in this chart. {% endhint %} {% hint style="success" %} The term "selfie PR" refers to pull requests that were merged by the author with no comments or reviews from others (excluding comments and reviews from bots) {% endhint %} {% hint style="success" %} **What good looks like** In Google's DORA research, the top 15th percentile have a `Change Failure Rate` of 0-4% [(see DORA 2025](https://dora.dev/research/2025/)). The DORA benchmark has changed over the years – it was[ 0%-5%](https://storage.googleapis.com/gweb-cloudblog-publish/images/1_-_elite_performer_graph.max-1400x1400.png) until recently, and [0%-15%](https://storage.googleapis.com/gweb-cloudblog-publish/images/Calculating_the_metrics_frOhcbp.max-2800x2800.jpg) before that. {% endhint %} # Mean Time to Recovery {% hint style="info" %} ⭐️ This metric is one of the [4 Key Metrics published by Google's DevOps Research and Assessment (DORA)](https://dora.dev/guides/dora-metrics-four-keys/) team. You can see all 4 key DORA metrics on the [DORA Metrics page](https://app.multitudes.co/DORA) of the Multitudes app. {% endhint %} {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Mean Time to Recovery graph](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/6621a6345425e2d43154d48c_MTTR.png) **What it is:** This is our take on [DORA's](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) `Mean Time to Recovery` metric. It's a measure of how long it takes an organization to recover from an incident or failure in production. You will need to [integrate with OpsGenie](/integrations/opsgenie) or [PagerDuty](/integrations/pagerduty) to get this metric. **Why it matters:** This metric indicates the stability of your teams’ software. A higher `Mean Time to Recovery` increases the risk of app downtime. This can further result in a higher `Change Lead Time` due to more time being taken up fixing outages, and ultimately impact your organization's ability to deliver value to customers. In [this study by Nicole Forsgren](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2681906) (author of DORA and SPACE), high performing teams had the lowest times for `Mean Time to Recovery`. The study also highlights the importance of organizational culture in maintaining a low `Mean Time to Recovery`. **How we calculate it:** We take a mean of the recovery times for the incidents that occurred in the selected date range, for the selected cadence (e.g. weekly, monthly). The line chart series are grouped by Multitudes team for Opsgenie, and Service or Escalation policy for PageDuty. The recovery time is calculated as follows:\ \ **On OpsGenie:** the time from when an incident was opened to when it was closed.\ \ **On PagerDuty:** the time from the first `incident.triggered` event\* to the first `incident.resolved` event. We attribute the incident to the team(s) of the resolver; this is the user who triggered the first `incident.resolved` event. This is how we determine whether to show an incident based on the team filters at the top of the page\*\*. \*If a trigger event can not be found, we default to the incident's created date. This is the case for historical data (the data shown when you first onboard). Also, in historical data, the resolver is assumed to be the user who last changed the incident status; you can't un-resolve an incident, so for resolved incidents this can be assumed to be the responder. \*\*If an incident was resolved by a bot, here's how they are shown in the data: * Incidents resolved by bot, with no assignee in its history: only shown when the `Teams` filter at the top of the page is set to showing the whole organization. * Incidents resolved by bot, with an assignee who is a Multitudes contributor: shown & attributed to the team(s) of that assignee. If there are multiple assignees, or there were multiple assignees throughout the history of the incident (e.g. it was reassigned), we take the last assignee(s)' team(s). * Incidents resolved by a Multitudes contributor: shown & attributed to the team(s) of the resolver. * Incidents resolved by a user who’s not a contributor: not shown. {% hint style="success" %} **What good looks like** In Google's DORA research, the top 15th percentile recover from failures in less than one hour [(see DORA 2025](https://dora.dev/research/2025/)). Note that DORA now tracks Failed Deployment Recovery Time (FDRT), which is very similar to Mean Time to Recovery. The difference is that FDRT only includes failures caused by a change the team made, whereas MTTR looks at recovery from all failures ([read more about the difference here](https://www.multitudes.com/blog/mttr-metrics)). We continue to track MTTR because many organizations still use it for reporting. {% endhint %} # Mean Time to Acknowledge {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Mean Time to Acknowledge](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/66988f3f756a23110d422c05_MTTA.png) **What it is:** This measures how long it takes an organization to acknowledge a new incident in production. You will need to [integrate with OpsGenie](/integrations/opsgenie) or [PagerDuty](/integrations/pagerduty) to get this metric. **Why it matters:** This metric indicates the responsiveness of your systems. A higher `Mean Time to Acknowledgement` increases the risk of app downtime, as it meant your teams and systems are taking longer to detect a failure in production. This results in a less reliable service or product for your customers. It can also impact flow of work elsewhere, since more time being taken up fixing outages, ultimately impacting your organization's ability to deliver value to customers. It is a subset of `Mean Time to Recovery`. **How we calculate it:** An incident's `Time to Acknowledgement` is calculated as the time from when an incident first fires off, to the first acknowledgement by a team member. See below for more details on how this is calculated for each IMS platform. {% hint style="info" %} For MTTA, the times are averaged over the selected date range, for each cadence (e.g. weekly, monthly). The line chart series are grouped by Multitudes team for Opsgenie, and Service or Escalation policy for PageDuty. Incidents that have not been acknowledged are not included in the data. This means that if many of your incidents are resolved without getting acknowledged, then your data may look sparse. {% endhint %} ### The Time to Acknowledgement is calculated as follows {% tabs %} {% tab title="On PagerDuty" %} Time to Acknowledgement = the time from the first `incident.triggered` event\* to the first `incident.acknowledged` event. We attribute the incident to the team(s) of the acknowledger. This is how we determine whether to show an incident based on the team filters at the top of the page\*\*. \*If a trigger event can not be found, we default to the incident's created date. This is the case for historical data (the data shown when you first onboard). Also, in historical data, the acknowledger is assumed to be the user who last changed the incident status. \*\*If an incident was acknowledged by a bot, here's how they are shown in the data: * Incidents acknowledged by bot, with no assignee in its history: only shown when the `Teams` filter at the top of the page is set to showing the whole organization. * Incidents acknowledged by bot, with an assignee who is a Multitudes contributor: shown & attributed to the team(s) of that assignee. If there are multiple assignees, or there were multiple assignees throughout the history of the incident (e.g. it was reassigned), we take the last assignee(s)' team(s). * Incidents acknowledged by a Multitudes contributor: shown & attributed to the team(s) of the acknowledger. * Incidents acknowledged by a user who’s not a contributor: not shown. {% endtab %} {% endtabs %} {% hint style="success" %} **What good looks like** From looking into the SLAs for P1 incidents of various organizations, and our own research on typical acknowledgements times within our own data, we've found that acknowledgement within 15 minutes of an incident being raised is a good target to aim for. {% endhint %} # Number of Pages {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Number of Pages graph](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/6621a653104d09e01a7e75d8_Number%20of%20pages%20pagerduty.png) **What it is:** This is the number of pages grouped by service or escalation policy. You will need to [integrate with PagerDuty](/integrations/pagerduty) to get this metric. **Why it matters:** This metric indicates the stability of your teams’ software. More pages means more incidents, and therefore disruptions to your team's focus. By looking at which services and escalation policies are generating the most pages, you can tune your monitoring to ensure that the pages you get are high signal. **How we calculate it:** We count the number of unique `incident.acknowledged` events that occurred within the selected date range. We then group by service or escalation policy, and stacked by urgency or priority based on the filter at the top of the Quality of Work page.\ ‍\ One incident may have multiple pages, and therefore multiple acknowledgements. These are counted separately.\ If an incident was resolved without getting acknowledged, we count the `incident.resolved` event as the acknowledgement for that incident. This metric is not available on historical data, because we don't have access to the incident's history of events. Incidents with the keyword `[test]` in the title (case insensitive) will not be processed. {% hint style="success" %} **What good looks like** Generally, fewer pages means fewer things getting broken in production! However, what's most important is that the pages you do get are indeed significant outages that are worth a wake-up call, and not false alarms. Click on the stacked bars to see if the pages look like signal, or noise. {% endhint %} # Deployment Failure Rate {% hint style="warning" %} Note: this metric is **only shown at a team level**, not an individual level. {% endhint %} ![Deployment Failure Rate](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/64927f6d16eed3fa15a901d1_Deployment%20Failure%20Rate%20\(1\).png) **What it is:** The percentage of [attempts to deploy to production](/metrics-and-definitions/deployment-metrics) that failed (or timed out). {% hint style="warning" %} Note this currently is only available if you’re using the [GitHub Actions integration](/integrations/github-actions). If you’re using the [Deployments API](about:/help/apis#deployments), for now we only receive successful deploy attempts. In the future, we may also accept indicators of failed attempts to calculate failure rate. {% endhint %} **Why it matters:** `Deployment failure rate` directly impacts the efficiency of your system. High failure rates can delay feature releases, bug-fixes and contribute to frustration and stress among developers. This can also divert developers attention away from focus work and contribute to distractions. Ideally, potential failures are caught in earlier testing environments, like `dev` and `staging`. If Deployment failure rate is high, it might mean there is not enough test coverage, or the checks you go through before triggering a prod deploy are missing some areas of the code. **How we calculate it:** This rate measures failed attempts to deploy to production (which includes time outs) divided by attempts that either were successful or failed, for clarity on how this is defined, review [here](/metrics-and-definitions/deployment-metrics). # People Metrics Learn about our wellbeing and collaboration metrics. We understand that productivity is about more than just speed and output. As [the recent paper on SPACE metrics](https://queue.acm.org/detail.cfm?id=3454124) points out, metrics signal what is important to an organization - and flow metrics alone cannot capture critical dimensions like employee satisfaction, well-being, retention, collaboration, and knowledge sharing. This is why we provide people metrics that look at well-being and collaboration, as well as our process metrics on flow of work, value delivery, and quality of work. ## Learn more about our People Metrics
Wellbeing/pages/j7pAbMu8DN4v2JpJg0w6
Collaboration/pages/Zp3FnJ9VUeiV1Vym10Ab
# Wellbeing Our key metrics to measure team wellbeing. In this group, we look at measures that reflect how well the people on a team are doing. Burnout is a huge issue in tech companies, with [60% of tech workers reporting that they’re burned out](https://www.teamblind.com/blog/index.php/2018/05/29/close-to-60-percent-of-surveyed-tech-workers-are-burnt-out-credit-karma-tops-the-list-for-most-employees-suffering-from-burnout/) – and the COVID pandemic has only exacerbated this. That’s why we look at indicators of how sustainably people are working and how well the work environment supports people to be healthy and well. ## Explore our Wellbeing metrics
Out-of-Hours Work/pages/0WHXVlD12xrtrBNQDM9z
Page Disruptions/pages/Lcirx3AlxaDvnBRg4RB2
Meeting Load/pages/TB4wR6difI4d8DK1mv4Q
### Filtering Wellbeing metrics All Multitudes metrics can be filtered by date range, team/s and repositories, and switched between daily, weekly, or monthly views to match your reporting needs. **If you use our PagerDuty integration, you can also filter your Incident metrics by either Priority or Urgency.** ![](/files/G2h3ojKOLNuZ8viDMFq1) **How it works:** * When [installing your PagerDuty integration](/integrations/pagerduty), your Multitudes owner will have selected your default filter setting based on whether it's most useful for you to filter incidents on urgency or priority level. * You can adjust this in your session by clicking the filter icon in the filter to switch between Urgency and Priority * You can then filter based on those levels This will impact the [Page Disruptions chart](/metrics-and-definitions/people-metrics/wellbeing/page-disruptions). # Out-of-Hours Work ![Out-of-Hours Work](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/634800eb26472ba9dde80333_Out%20of%20Hours%20Work.jpg) ‍**What it is:** This measure shows how often people are working outside of their own preferred working hours. Given that more and more people are working flexible hours, our metric is configurable for different timezones and different preferred working hours and days.**‍** ‍**Why it matters:** Working long hours is a [risk factor for burnout](https://www.researchgate.net/publication/302591873_The_Associations_Between_Long_Working_Hours_Physical_Inactivity_and_Burnout#:~:text=Conclusions%3A%20Long%20working%20hours%20are,reduce%20the%20risk%20of%20burnout.). Moreover, the longer someone works, the harder it is for them to solve challenging problems: [a study from the Wharton School of Business and University of North Carolina](https://hbr.org/2015/04/its-the-weekend-why-are-you-working) demonstrated that our cognitive resources deplete over time, so we need breaks to refuel. At Multitudes, we’ve seen that the faster a team’s `Change Lead Time`, the higher their `Out-of-Hours Work` is likely to be – so it’s important for teams and leaders to keep an eye on both metrics together, so they don’t over-optimize for speed and then burn out their team.**‍** **How we calculate it:** We look at the number of commits that people did outside of their usual working hours. By default, this is set to 8am-6pm, Monday to Friday, in each team member’s local time. This can be individually configured in Settings to account for different working hours and days. {% hint style="success" %} **What good looks like** On average over time, this should be 0, with people doing as little work out of hours as possible. If this does rise above 0, it’s important to ensure that it doesn’t become a trend so that people aren't doing sustained days of long hours. Multiple weeks with someone doing more than 5 commits made out-of-hours per week might warrant some rebalancing of work or stricter prioritization! {% endhint %} # Page Disruptions
Chart showing Team Yetis had the most disruptions in the past 6 weeks.

Page Disruptions

**What it is:** The number of pages over time. You can see `Out-of-hours pages` (the ones outside of the responder’s preferred working hours\*, which can contribute to burnout), or `All pages` (includes pages during work hours, whch are still disruptive to a team's focus). *💡 Use our in-chart toggles to view these based on either the number of pages, or the number of hours disrupted by pages.* {% hint style="info" %} \*By default, this is set to 8am-6pm weekdays, local time. Given that more and more people are working flexible hours, our metric is configurable for different timezones and different preferred working hours and days on each team member's profile in Settings. {% endhint %} **You will need to** [**integrate with PagerDuty**](/integrations/pagerduty) **to get this metric.** **Why it matters:** Being woken up in the middle of the night is never fun. Continued disruptions to sleep impact people's productivity, wellbeing, and satisfaction in their workplace. What's more, lots of pages disrupting a team (no matter the hour) can interrupt delivery. **How we calculate it:** We use the time of acknowledgement as a proxy for when the page was sent out (when the responder was disturbed). We count the number of acknowledgements on incidents, using the time that each `incident.acknowledged` event occurs\*. We then compare this time to the responder's preferred working hours, and figure out if it was an out-of-hours page. If multiple people acknowledged the incident on PagerDuty, the chart will count all acknowledgements. For example, if Person A acknowledges an incident, then reassigns to Person B who also acknowledges it, both Person A and Person B (and their respective teams) will have +1 to their Page Disruptions on this chart. \*If an incident was resolved without getting acknowledged, we count the `incident.resolved` event as the acknowledgement for that incident. {% hint style="info" %} This metric is not available on historical data, because we don't have access to the incident's history of events. {% endhint %} Incidents with the keyword `[test]` in the title (case insensitive) will not be processed. **Disrupted hours:** This view shows the number of distinct calendar hours disrupted by pages. \ For example, if there are pages at 6.30pm, 7.15pm and 7.25pm we would count **two disrupted hours** (6-7pm, and 7-8pm). Viewing the disrupted hours accounts for the difference (and human impact) between being paged 10 times in one hour for an incident vs five pages over five distinct hours. {% hint style="success" %} **What good looks like** Generally, the less pages means the more reliable your tech stack, and the less disruptions to workflow. If the number rises above 1-2, especially for out-of-hours pages, it’s important to ensure that it doesn’t become a trend so that people aren't doing sustained days of long hours and interrupted sleep. Multiple weeks of someone being paged out-of-hours might indicate some QA process changes or reliability work is needed. {% endhint %} # Meeting Load **What it is:** The number of hours spent in meetings, on average for a team and for individuals. You can view either Out-of-hours meetings (meetings outside of the individual’s preferred working hours), or All meetings (which includes meetings during and outside of working hours). You will need to [integrate with Google Workspace](/integrations/google-calendar) to get this metric.**‍** **Why it matters:** [78% of people surveyed by Atlassian](https://www.atlassian.com/blog/workplace-woes-meetings) said they’re expected to attend so many meetings that it’s hard to get their work done. This leads to people working overtime to make up for time spent in meetings and feeling more drained at the end of a meeting-heavy day. **How we calculate it:** We find all eligible meetings in each person’s calendar and calculate their time based on the start and end time. **We exclude the following events from our categorisation of meetings:** * Events with a single attendee - this means that if you use your calendar for time blocking or personal reminders, that won’t affect our calculations. * Events that are over 12 hours - as these are usually placeholders for things like strategy days * Events marked as "free" or "maybe"/"tentative" - even if multiple people have accepted the event invite * Non-meeting event types, such as 'focus time' If two eligible meetings overlap, we count them both as meeting hours in the Meeting Load chart. This may result in a higher than expected count of hours. For example, two overlapping 1 hour meetings would result in a Meeting Load count of 2 hours. {% hint style="info" %} Note that this is different from our approach with [`Focus Time`](/metrics-and-definitions/process-metrics/flow-of-work/focus-time), because here we’re focusing on meeting burden. Although an individual can only be in one meeting at a time, this approach reflects the load placed on a team member when they are expected in overlapping meetings, as this creates additional work for them to figure out which meeting to attend and communicate that to the organizers. {% endhint %} For the Out-of-hours meetings calculation, we look at meetings that happened outside of someone’s working hours. By default, working hours are set to 8am-6pm weekdays, local time. To support flexible hours, our metric is configurable for different time zones and different preferred working hours and days on each team member's profile in [Settings](https://app.multitudes.co/teamSettings). Changes to your working hours in Settings will flow through to the Meeting Load analysis. {% hint style="info" %} **How we handle Group Meetings in Outlook:** * If you and 1+ other person are both marked “Busy”, then it’s a meeting. * If the event is Free/ Working Remotely / Tentative / Out of office, then we don’t count it as a meeting {% endhint %} **How we handle private events** Events marked as "Private" will still be counted in your Meeting Load analysis but the Event Title and Attendees will be redacted.
We do not currently support private calendars. If your organisation has private calendar settings (where all members of the organisation can only see free/busy instead of details) you can redact **all event details** from Multitudes. Contact to enable this setting for your organisation. {% hint style="success" %} **What good looks like** We know some meetings are necessary, but the more meetings people have, the harder it is to get other work done, and [they may end up feeling drained](https://www.atlassian.com/blog/workplace-woes-meetings). That’s why we recommend no more than 15 hours/week – or 3 hours/day – of meetings for your team members. Out-of-hours meetings can be hard to avoid for globally-distributed teams. However, it’s important to keep an eye on so that people don’t continuously have to join meetings that stretch out their work days or disrupt their sleep. Here, we recommend no more than 2 hours/week of out-of-hours meetings – so that your teams can do a meeting or two with people in other timezones, but not more than that. {% endhint %} # Collaboration Learn how feedback and collaboration patterns flow through your team. ## Explore our Collaboration metrics
PR Participation Gap/pages/J72HcpGSvHwq1Db8q1na
PR Feedback Given/pages/Dl2RBE2f52QzcusbBDTW
PR Feedback Received/pages/lrMOM844tAhdqyNa090Y
Feedback Flows/pages/Rr5LCL69bb2ro7VDlSSh
Feedback Quality/pages/L8a0QwpA3J98dd0kUIXt
Feedback Themes/pages/9KksK196cFHWsOw7djX7
### Filtering Collaboration metrics All Multitudes metrics can be filtered by date range, team/s and repositories, and switched between daily, weekly, or monthly views to match your reporting needs. By default, all metrics on the Collaboration page are based on comments given. You can use the **Show Reviews Only** toggle to switch this to show reviews with at least 1 comment, instead of showing comments. This excludes reviews with no comments and reviews made after a PR was approved. # PR Participation Gap ![A stylized line graph shows time to merge is high at 18 hours but is trending up.](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/634800eb51d56a1a8ed35992_Participation%20Gap.jpg) **What it is:** This shows the absolute difference between the most and least frequent commenters on the team.**‍** **Why it matters:** This measure shows how imbalanced team participation is in reviews and comments. More balanced participation is a behavioral indicator of psychological safety, which [Google’s Project Aristotle research](https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html) showed is the number one determinant of team performance. **How we calculate it:** We count the **number of comments that each person has written and then show the range** from the highest count to the lowest count. * We exclude team members who wrote zero comments, because sometimes teams will have a few team members who are not on GitHub often, but included in the data. * We can only calculate this for teams with at least 2 people; for a team of one person, there is no gap to calculate. * We include comments on all PRs, including draft and open PRs. This is because it’s still someone giving feedback / sharing knowledge with someone else, even if the PR is still in a draft state. {% hint style="success" %} **What good looks like** The smaller the gaps are, the better – a smaller gap means that people are contributing more equally. Looking at distributions of participation gaps each week across various teams and organizations, we found that a threshold difference of 25 comments would be a reasonably realistic goal for most teams. {% endhint %} # PR Feedback Given ![A stylized line graph shows time to merge is high at 18 hours but is trending up.](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/634800ee6be9560009ae8ec5_PR%20Feedback%20Given.jpg) **What it is:** The number of comments written on PRs. **Why it matters:** This visualizes who is giving the most support, since PR reviews and comments are a way to share knowledge and to encourage growth and learning opportunities.**‍** Giving feedback on PRs can be an example of glue work, the somewhat-invisible work that people do to lift up others on the team; our goal is to make this work more visible and valued on teams. **How we calculate it:** The **total number of comments written on PRs**, including comments on one's own PR. We include comments on your own PR because they are often in response to a reviewer question, so these can also contribute to learning and knowledge-sharing on the team. {% hint style="success" %} **What good looks like** While written communication styles differ between individuals, if a team that does their code reviews on GitHub, then 10 comments per person is a good benchmark to hit. This is based on research from our own data, looking across 6 person-weeks of data for 10 randomly sampled orgs in the Multitudes dataset. {% endhint %} {% hint style="info" %} The trends we expect will vary by level. Senior engineers are expected to give more feedback than juniors, to share their knowledge across the team. However, juniors have a lot to offer in code reviews too, via a fresh perspective and clarifying questions (more here about [why it’s important to include juniors in code reviews](https://shortcut.com/blog/why-you-should-always-involve-junior-developer-in-code-review)). That’s why we still recommend teams aim for more balanced participation across the team – it’s always good to make sure that your juniors feel comfortable speaking their mind and asking questions during code review. {% endhint %} # PR Feedback Received ![A stylized line graph shows time to merge is high at 18 hours but is trending up.](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/634800eba996cf92d3ee0bb5_PR%20Feedback%20Received.jpg) **What it is:** The number of comments received on PRs. **Why it matters:** [Research](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ICSE202013-codereview.pdf) shows that code review is important for knowledge-sharing and collaborative problem-solving; this metrics helps you ensure that everyone in the team is receiving enough support and feedback that they need. While this is crucial for juniors, continual learning and growth matters for seniors too. For an example, see [this success story](https://www.multitudes.co/success-stories/storypark) on how one of our customers increased how much feedback seniors were getting from their peers.In addition, there’s also [bias in who gets good feedback](https://fortune.com/2014/08/26/performance-review-gender-bias/). Specifically, people from marginalized groups are more likely to get less, and lower-quality feedback. This is why it's important to have data to make sure everyone on the team is getting the support. **How we calculate it:** The **total number of comments written on the PRs that you've authored, excluding comments you've written on your own PR** (since you don't give feedback to yourself).**‍** {% hint style="success" %} **What good looks like** Similarly to `PR Feedback Given`, our benchmarks show that it’s good to aim for at least 10 comments per week to each person on the team. This is based on research from our own data, looking across 6 person-weeks of data for 10 randomly sampled orgs. {% endhint %} {% hint style="info" %} There are nuances – for example, juniors might receive more feedback than seniors. We recommend you use this data to focus on outliers. Someone getting very little feedback might not be getting enough support on their work. Someone getting lots of feedback might feel overwhelmed or could be the target of nitpicking. {% endhint %} # Feedback Flows
Sankey chart showing feedback flows between 5 team members.

A sankey chart showing flows of feedback between team members

**What it is:** This graph shows how much feedback each person gave on other people’s PRs, how much feedback they got on their own PRs, and how feedback flows between people. If you have[ levels set up](/configuration-and-setup/configuring-your-team), the graph will be color-coded by level, which can help you quickly see at-a-glance if feedback flows are as expected across your team. **Why it matters:** The [top benefits of code reviews](https://smartbear.com/state-of-software-quality/code-review/) are improving code quality, knowledge-transfer, and learning. Moreover, there’s [bias in who gets good feedback](https://fortune.com/2014/08/26/performance-review-gender-bias/). Visualizing feedback flows can show us whether there are silos, and how we’re doing across the team at supporting each other. **How we calculate it:** We look at the number of comments and reviews that each person (or team) gave and received on their PRs. We then show how the feedback moves across people and teams. {% hint style="success" %} **What good looks like** In the best teams, everyone is giving feedback and everyone is receiving feedback, or at least asking questions about others’ work. In these teams, members in senior levels give plenty of feedback to juniors and intermediates – and juniors and intermediates feel comfortable asking questions to seniors. {% endhint %} We also look at several indicators of collaboration. In this bucket, we’re examining who gets support and who’s not getting enough support. We also show the people who are doing a lot of work to support others. This type of [“glue work”](https://noidea.dog/glue) is easy to miss but is important for team success and [benefits the whole organization](https://hbr.org/2018/07/why-women-volunteer-for-tasks-that-dont-lead-to-promotions). These metrics show patterns in comments on GitHub. To see review patterns, you can turn on the `Show reviews only` filter; this will show only reviews with at least 1 comment, rather than all comments. # Feedback Quality Learn how we use AI and machine learning to surface insights about the quality of feedback in your team's code reviews.
Stacked bar chart showing percentage of feedback given on PRs grouped by specific, neutral, unspecific, and minimal review.
### What it is This shows the overall quality of feedback given in code reviews. We analyze all code review comments (excluding the PR author's own comments) and classify them into quality categories based on how constructive and actionable the feedback is. ### Why it matters The quality of feedback in code reviews directly impacts team psychological safety (see [Belschak et al.](https://iaap-journals.onlinelibrary.wiley.com/doi/10.1111/j.1464-0597.2008.00336.x)), [learning outcomes](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ICSE202013-codereview.pdf), and of course, [code quality](https://smartbear.com/state-of-software-quality/code-review/). Research consistently shows significant feedback gaps across different groups in the workplace (see [Gunawardena et al.](https://kblincoe.github.io/publications/2022_CSCW_Destructive.pdf)). According to [Lean In and McKinsey & Co](https://wiw-report.s3.amazonaws.com/Women_in_the_Workplace_2016.pdf), women are more than 20% less likely than men to say their manager gave them critical feedback that contributed to their growth. Additionally, a [recent survey by Textio](https://textio.com/feedback-bias-2024) found that Black and LatinX employees are more likely to receive feedback about their "personality" than the actual quality of their job performance. Understanding your team's feedback quality helps you create more inclusive review processes and identify opportunities to improve collaboration. ### How we calculate it We analyze feedback given in code reviews using Multitudes's AI models that have been specifically designed to mitigate algorithmic biases and are grounded in research. First, we find each set of feedback – we group all the comments that a commenter did on someone’s PR into one set. The reason we do this is because it’s typical for the first review on a PR to be more detailed than follow-up reviews, so bringing in the full set of comments gives our model more context.
We then classify feedback into **four quality categories:** * **Highly Specific:** Detailed, actionable feedback that clearly explains what needs to change and provides clear reasoning. This also includes thought-provoking comments that share knowledge or offer alternative approaches. * **Neutral:** Moderately detailed feedback that provides some guidance but could be more comprehensive * **Unspecific:** Vague comments that don't provide clear direction for improvement * **Minimal:** Short reviews like "LGTM :thumbsup:" or "Shippit :fire:!!" that provide minimal guidance. These reviews are commonly referred to as "rubber-stamp" code reviews. *Note: PR Feedback where it was a code review with no comments are not included in this.* * **Needs Attention:** Feedback that has been classified as `Negative` by our model and so may come across as harsh, dismissive, or potentially harmful. We classify as "Needs Attention" rather than "Negative" to acknowledge that a model will always lack the context a human will. We look at patterns across your team's code review conversations to surface insights about collaboration dynamics and feedback culture. If you notice any incorrect model classifications please provide feedback using the 🚩button in the drill down table. We also support exclusion rules where you can configure terms to be excluded (see [#exclusion-terms](#exclusion-terms "mention") below for more). #### **Understanding Feedback that "Needs Attention"** When feedback is classified as "Needs Attention", it is because a model believes it to be `Negative` – so we recommend a human take a look to bring more context. For transparency, we also identify the specific reasons for the model classification based on established research on what makes code review feedback destructive:

Research and Insights into Negative and Destructive Criticism

#### **Negativity Reasons:** * **Personal Attack:** Feedback that targets the person rather than the code * **Vague Criticism:** Critical feedback without clear suggestions for improvement * **Judgmental:** Feedback with a judgmental or condescending tone * **Harsh Language:** Use of inconsiderate or unnecessarily harsh language * **Excessive Nitpicking:** Repeated focus on minor issues without addressing bigger picture concerns * **Negative Emojis:** Overuse of negative emojis that create a hostile tone * **Terseness:** Overly brief feedback that comes across as dismissive This granular analysis helps teams understand not just that negative feedback occurred, but what specific patterns to address in their code review culture. ### **What good looks like** Feedback quality patterns vary significantly based on team dynamics, code review practices, and organizational culture. That said, based on our initial analysis across Multitudes customers, teams typically have 15% highly specific feedback, 20% unspecific feedback, 25% minimal feedback and less than 2% negative feedback classified as "Needs Attention". We recommend teams aim for: * 20%+ highly specific feedback because this provides clear, actionable guidance * Zero negative feedback because this can be destructive for team inclusion and performance. Feedback which our models consider may be potentially negative is classified as "Needs Attention" on the chart. * <30% minimal feedback, because some quick approvals are normal, but excessive rates suggest insufficient review depth. * Equitable distribution across the team, with all team members receiving similar quality of feedback and no one getting significantly less specific feedback.” Use these insights in 1:1s, retros, and team discussions to foster a more supportive and effective code review culture. {% hint style="info" %} Multitudes is actively conducting research to identify what percentages of feedback quality correlate with high-performing teams. As we gather more data and insights, these benchmarks will be updated to provide more precise targets for healthy feedback patterns. Learn more about our [original research here](https://www.multitudes.com/research). {% endhint %} ### Exclusion Terms You can configure terms to be excluded from the Feedback Quality chart through the exclusion rules. This is particularly useful for filtering out bot commands, automated messages, or other non-human feedback that shouldn't be analyzed. To add or modify exclusion terms, click the write icon on the chart header:
Pointer on chart over edit button, with tooltip text saying "Edit rules to exclude specific terms from this analysis"

Exclusion rules apply only to complete matches, not partial text, and changes take up to 6 hours to apply. When you add a new exclusion term, we will apply it to future data; if you need to exclude terms from existing historical data, contact . ### Provide Feedback on Model Predictions We're continuously improving our Multitudes AI models to ensure accurate classification. If you've noticed a comment is misclassified — such as constructive feedback labeled as `Needs Attention` or vague comments marked as `Highly Specific`, please use the :triangular\_flag\_on\_post: (the red flag) button in the drill down table to report it. Your feedback helps us refine our models and reduce algorithmic bias, ensuring better insights for your teams. We review all flagged predictions and use it to make the feedback quality analysis more reliable for everyone using Multitudes. Note that our language model does better with domain-specific knowledge in English than in other languages, so we’d especially love to hear from Multitudes users who write reviews regularly in other languages. ### Research on Feedback Quality and Code Review The design of this feature is informed by extensive research on feedback quality and code review dynamics and alongside [collaboration with our academic partners](https://www.multitudes.com/research). Thanks to our original research in this space, we've now published a peer-reviewed paper with our findings on feedback quality – see highlights in this [one-pager](http://bit.ly/FSE-one-pager), or [view the full preprint](https://kblincoe.github.io/publications/2026_FSE_Multitudes.pdf). Some key studies and articles we recommend users read include: * [Destructive Criticism in Software Code Review Impacts](https://kblincoe.github.io/publications/2022_CSCW_Destructive.pdf) * [Predicting developers' negative feelings about code review](https://dl.acm.org/doi/10.1145/3377811.3380414) * [Unlearning toxic behaviors in a code review culture](https://medium.com/@sandya.sankarram/unlearning-toxic-behaviors-in-a-code-review-culture-b7c295452a3c) * [Expectations, outcomes, and challenges of modern code review](https://ieeexplore.ieee.org/document/6606617) * [Negative Effects of Destructive Criticism: Impact on Conflict, Self-Efficacy, and Task Performance](https://pubmed.ncbi.nlm.nih.gov/3384772/) * [Process Aspects and Social Dynamics of Contemporary Code Review: Insights from Open Source Development and Industrial Practice at Microsoft](https://ieeexplore.ieee.org/document/7484733) * [Detecting interpersonal conflict in issues and code review: cross pollinating open- and closed-source approaches](https://dl.acm.org/doi/10.1145/3510458.3513019) * [Towards Unmasking LGTM Smells in Code Reviews: A Comparative Study of Comment-Free and Commented Reviews](https://ieeexplore.ieee.org/document/10794990) # Feedback Themes Learn how we use AI and machine learning to surface insights about common topics that appear in your team's code reviews.
Stacked bar chart showing percentage of feedback given on PRs grouped by themes like Code Quality, Code Structure, Documentation and Risk/Error Mitigation.
### What it is This shows the feedback themes that appear in code reviews. We analyze all code review comments (excluding the PR author's own comments) and classify them into themes. Within each theme, we also classify each set of feedback into an underlying code, which provides more specificity about what the feedback covered. The themes and codes are grouped together in an underlying thematic map – read below for more about that. ### Why it matters Identifying common themes in our code reviews can help us identify systemic areas of improvement across our engineering team and processes, and it can help us identify coaching opportunities for specific team members. We all know how important code reviews are for good outcomes – when they're done well, they're linked to: * Better codebases: Higher [code quality](https://smartbear.com/state-of-software-quality/code-review/), a direct impact on [ease of maintenance](https://dl.acm.org/doi/10.1145/2884781.2884840), [fewer defects, and more maintainable-code](https://www.researchgate.net/publication/372762289_The_Effectiveness_of_Code_Reviews_on_Improving_Software_Quality_An_Empirical_Study) * Happier people: [More knowledge-sharing, more learning](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ICSE202013-codereview.pdf), [more psychological safety](https://iaap-journals.onlinelibrary.wiley.com/doi/10.1111/j.1464-0597.2008.00336.x), and [less turnover](https://cmustrudel.github.io/papers/seis22pushback.pdf) * More efficient processes: [less waste during development](https://www.researchgate.net/publication/339026024_Knowledge_Sharing_Factors_for_Modern_Code_Review_to_Minimize_Software_Engineering_Waste) However, [poor-quality code reviews can actually introduce more bugs](https://plg.uwaterloo.ca/~migod/papers/2015/icsme15-OleksiiOlgaLatifa.pdf) – contrary to one of the main motivations for conducting reviews in the first place. ### How we calculate themes We analyze feedback given in code reviews using Multitudes's AI models that have been specifically designed to mitigate algorithmic biases and are grounded in research. Our approach builds on the methodology shared by [Mathias et al](https://pubmed.ncbi.nlm.nih.gov/39067136/) — of applying thematic analysis (see [Braun et al.](https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa)) with language models. **Our two-step classification process:** 1. First, our model identifies high-level themes in code review feedback (note: one unit of feedback can consist of multiple themes) 2. Then, we use these themes as context to predict specific research codes—the detailed topics discussed in your team's reviews. The predicted codes are visible in the drill down tables, which also show examples of each theme and code in your team's code reviews (see below). Within the drill-down the codes are ordered by how much of the feedback is related to the predicted code. We also exclude all `Minimal Reviews` from our analysis. If you want to see the amount of `Minimal Review` happening, refer to our [Feedback Quality feature](https://docs.multitudes.com/metrics-and-definitions/people-metrics/collaboration/feedback-quality).
Stacked bar chart showing percentage of feedback given on PRs, showing the person has received the most feedback on “Risk/Error Mitigation”.
Altogether, our model classifies the set of themes and codes that appear in each code review. The model selects these from 15 themes and 45 research codes which are linked together via a thematic map. **What is a thematic map?** In qualitative research, a thematic map is a visual representation of the relationships between different themes and codes derived from qualitative data. It's a way to organize and display the key ideas and their connections, helping researchers to understand and interpret the data more effectively. To learn more, read the original paper by [Braun et al.](https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa)
**How do we measure model performance?** Because this model outputs are both hierarchical and multi-label, evaluation and accuracy metrics aren't straightforward. We implemented several complementary evaluation metrics: * Basic classification metrics (F-score, precision, recall at each hierarchy level) * Wu-Palmer similarity and edge-distance metrics (accounting for hierarchical relationships between themes) We prioritize minimizing cross-theme errors (e.g., labeling "Security" as "Documentation") over within-theme errors (e.g., "Code Optimization" vs "Code Readability"). All metrics are validated against our human-labeled ground truth dataset which was labelled by domain experts. Some examples of themes and code that our model can classify your code reviews as containing: * **Code Structure:** Code Organization, Code Reusability * **Risk/Error Management:** Error management, Edge Case Handling, Technical Debt Management, Bug Prevention * **Knowledge Sharing and Growth:** Knowledge Sharing, Implementation Suggestions * **Deployment Ops:** Deployment Process, Environment Configuration, Release Management, Dependencies * **Code Quality:** Code Redundancy, Code Simplification, Code Fix, Code Optimization {% hint style="info" %} Multitudes is actively conducting research to identify what breadth of topics present in feedback themes correlates with high-performing teams. As we gather more data and insights, these benchmarks will be updated to provide more precise targets for healthy feedback patterns. Learn more about our [original research here](https://www.multitudes.com/research). {% endhint %} ### Provide Feedback on Model Predictions We're continuously improving our Multitudes AI models to ensure accurate classification. If you believe that your feedback has been misclassified — for example, reviews being mislabeled for an incorrect different theme or code – please use the :triangular\_flag\_on\_post: (the red flag) button in the drill down table to report it. Your feedback helps us refine our models and reduce algorithmic bias, ensuring better insights for your teams. We review all flagged predictions and use it to make the feedback themes analysis more reliable for everyone using Multitudes. ### Research on Feedback Themes and Code Review The design of this feature is informed by extensive research on feedback themes and code review dynamics and alongside collaboration with our academic partners (see [Multitudes Research](https://www.multitudes.com/research)). Some key studies and articles we recommend users read include: * [Using thematic analysis in psychology](https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa) * [Modern Code Review: A case study at Google](https://research.google/pubs/modern-code-review-a-case-study-at-google/) * [Investigating Code Review Quality: Do People and Participation Matter?](https://plg.uwaterloo.ca/~migod/papers/2015/icsme15-OleksiiOlgaLatifa.pdf) * [The Impact of Peer Code Review on Software Maintainability in Open-Source Software: A Case Study](https://thesai.org/Publications/ViewPaper?Volume=13\&Issue=12\&Code=IJACSA\&SerialNo=110) * [Knowledge Sharing Factors for Modern Code Review to Minimize Software Engineering Waste](https://www.researchgate.net/publication/339026024_Knowledge_Sharing_Factors_for_Modern_Code_Review_to_Minimize_Software_Engineering_Waste) * [The Effectiveness of Code Reviews on Improving Software Quality: An Empirical Study](https://www.researchgate.net/publication/372762289_The_Effectiveness_of_Code_Reviews_on_Improving_Software_Quality_An_Empirical_Study) * [Code review quality: how developers see it](https://dl.acm.org/doi/10.1145/2884781.2884840) * [Tools, processes and factors influencing of code review](https://www.researchgate.net/publication/344675044_Tools_processes_and_factors_influencing_of_code_review)
Our [**Feedback Quality**](https://docs.multitudes.com/metrics-and-definitions/people-metrics/collaboration/feedback-quality) feature was also informed by complementary research to do with code review processes and standards – you can read more about those in that feature's documentation linked [here](/metrics-and-definitions/people-metrics/collaboration/feedback-quality). # Deployment Metrics Deployment metrics (e.g., `Change Lead Time` if you’d like it to include deploy time, `Deploy Time`, `Deployment Frequency`, `Deployment Failure Rate`) require you to either install our [GitHub Actions integration](/integrations/github-actions) or use our [Deployments API](/integrations/deployments-api) for other CI/CD tooling. ## **If you’re using our Deployments API** An “attempt to deploy to production” refers to the POST that you’ve sent to our API. For timestamp, we’ll either use: * The optional deployedAt value if provided, OR * The timestamp when we receive your POST call When naming the deployment, we’ll either use: * The optional title value if provided, OR * The commit message of the matched commit with the most recent commit timestamp {% hint style="info" %} Note that we include everything. For example, if you send us 3 deployments with the same commitSha (let's say because you'd deployed it to dev, staging, and prod), we will show all three in the [Deployment Frequency](/metrics-and-definitions/process-metrics/value-delivery/deployment-frequency) chart. That said, if we receive multiple deployments with the same commitSha, metrics relating to time (e.g., [Deploy Time](/metrics-and-definitions/process-metrics/flow-of-work/deploy-time)) will only look at the latest attempt. {% endhint %} At the moment, there is not a way to delete specific deployments that you've sent to us. Please email [support@multitudes.com](mailto:support@multitudes.com?subject=Error%20in%20deployment%20POST) if a POST was made in error. ## **If you’re using our** [**GitHub Actions integration**](/integrations/github-actions) The definition of an “attempt to deploy to production” will depend on how you’ve set up GitHub Actions, so be sure to configure your GitHub Actions integration. 1. **If you selected "Workflows", this will be an attempt of any of the workflow runs that you've selected in your configuration** * If you have multiple workflows that deploy to prod, all deploys will be considered in your Multitudes metrics. Here are some examples: * If you have two workflows, and one fails and one succeeds: 1. The failed workflow gets counted as part of `Deployment Failure Rate` 2. The Successful workflow gets counted as part of `Deployment Frequency` 3. `Change Lead Time` will use the successful workflow run 4. `Deploy Time` will use the successful workflow run * If you have two workflows, and both succeed: 1. `Change Lead Time` will use the earliest successful workflow 2. Both will be counted as part of `Deployment Frequenc`y 3. Both will be measured in `Deploy Time` 2. **If you selected "Environment/Deployments", this further depends on the metric** * For `Change Lead Time` (when including deploy time), `Deploy Time`, and `Deployment Failure Rate`, this will be an attempt of the workflow run that *contains* the deployment to the environment/s that you've selected in your configuration * If a deployment job successfully deploys code to your selected environment, but something else causes the overall workflow run to fail, then this will *not* count as a successful deploy * In this case, you’ll see a longer `Deploy Time` and `Change Lead Time`, since it’s waiting for another successful attempt of both deployment job and overall workflow run * For `Deployment Frequency`, this will be an attempt of just the deployment job itself to the environment that you've selected in your configuration * If a deployment job successfully deploys code to your selected environment, but something else causes the overall workflow run to fail, then this *will* still count as a successful deploy * Here's a concrete example of these nuances: * You have a workflow called `deploy-code`, which includes these jobs 1. `build-code` 2. `apply-terraform`, which works on the prod Environment you selected in your configuration step, and therefore represents the point at which your change is now shipped to users) 3. `send-update-to-slack`, an extra internal notification step * To deploy to production, you run `deploy-code`, and the jobs `build-code` and `apply-terraform` succeed. Your code is now out to users in the production environment. * However, `send-update-to-slack` fails * This *will* count toward `Deployment Frequency`, because your code actually did get deployed to production * This will *not* count toward `Deploy Time`, or `Change Lead Time` through to deployment, and will register as a failure in `Deployment Failure Rate` because the job that failed still disrupts the smooth flow of CI/CD (i.e., developers have to investigate) Lastly, to align on the definitions of success and failure: * Successful attempt = attempt concluded with `SUCCESS` * Failed attempt = attempt concluded with `FAILURE` , `TIMED_OUT` , or `STARTUP_FAILURE` * We ignore other conclusions (i.e., `CANCELLED` or `SKIPPED`) Note that these definitions apply regardless of the nuance expressed earlier across what an attempt is based on how GitHub Actions is used and configured. # Github How to connect GitHub to Multitudes. ## GitHub + Multitudes Integration Benefits * Integrating with GitHub is a prerequisite for using Multitudes * Select GitHub teams/team members to get contributor data in Multitudes * See a range of insights include [DORA](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) and [SPACE](https://queue.acm.org/detail.cfm?id=3454124) metrics. ## How it works ### GitHub Data Multitudes pulls in code metadata from GitHub; our app does not ingest your codebase. This metadata includes information about pull requests – such as when they were created, who the author was, commits made on the pull request, number of lines changed, etc. and about comments – including comments written on pull requests and reviews submitted. This metadata informs Multitudes metrics across [Flow of Work](/metrics-and-definitions/process-metrics/flow-of-work), [Value Delivery](/metrics-and-definitions/process-metrics/value-delivery), [Quality of Work](/metrics-and-definitions/process-metrics/quality-of-work), [Wellbeing](/metrics-and-definitions/people-metrics/wellbeing) and [Collaboration](/metrics-and-definitions/people-metrics/collaboration). When you set up the GitHub installation, you can see the full list of permissions that Multitudes requests. ### Teams and people Multitudes also gets some data from teams about their structure, e.g., which teams people are on, their roles, and their levels. This is used to populate your [Multitudes teams](/configuration-and-setup/adding-users-and-teams) and associate data with contributors. You can enable [GitHub Teams Sync](/configuration-and-setup/adding-users-and-teams#automatically-adding-team-members-via-github-teams-sync) to automatically update your Multitudes contributors based on GitHub teams. ## Requirements **Required role:** [Organization Owner](https://docs.github.com/en/organizations/managing-peoples-access-to-your-organization-with-roles/roles-in-an-organization#about-organization-roles) **Notes:** You must have admin access to select repositories and Github teams/contributors. ## How to install Follow the GitHub installation steps when you onboard to Multitudes. When installing, you will select which users/teams and repositories to integrate with Multitudes. ### How to add additional repos 1. Check what existing repos are synced at the top of any deep-dive page, e.g.,
App header showing filters, with Repositories filter highlighted.
2. If you need additional repos, add them via our GitHub App – go to [Integrations](https://app.multitudes.co/team-settings/integrations), click Configure, and follow the flow to GitHub.
Integrations page in Multitudes app showing GitHub card with Configure button.
3. The repos will sync for all data moving forward. To get 6 weeks of historic data, contact with a list of the repos you added and we’ll trigger a historic data pull. # Deployments API How to use Multitudes' API endpoints to track deployments and integrate with your CI/CD pipeline. ## Deployments API Benefits Use our API to get more flexibility and control over how you track your devOps activity. * See [`Deploy Time`](/metrics-and-definitions/process-metrics/flow-of-work/deploy-time) - how long code takes to get deployed once a PR has been merged and include Deploy Time in [`Change Lead Time`](/metrics-and-definitions/process-metrics/flow-of-work/change-lead-time) calculations * See how often code is successfully deployed to Production with [`Deployment Frequency`](/metrics-and-definitions/process-metrics/value-delivery/deployment-frequency) ([a key DORA metric](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance)) * Keep track of [`Deployment Failure Rate`](/metrics-and-definitions/process-metrics/quality-of-work/deployment-failure-rate) {% hint style="info" %} You can use *either* our Deployments API *or* our [GitHub Actions integration](/integrations/github-actions) to provide us with data about deployments. If you already have our GitHub Actions integration configured, generating a token to connect to our Deployments API will override the data from GitHub Actions. {% endhint %} ## How it works [See here for details](https://docs.multitudes.com/metrics-and-definitions/deployment-metrics#if-youre-using-our-deployments-api) on how data received via API is used to calculate Deployment metrics in Multitudes. ## Deployments Send us information about your deployments from any CI/CD tool. We’ll use this to calculate metrics such as `Change Lead Time` through to deployment and `Deployment Frequency` (more info on [Metrics & Definitions](/metrics-and-definitions/multitudes-insights)). {% hint style="warning" %} Our data pipeline runs every 6 hours so you may need to wait to see data after you start sending. {% endhint %} ## Authentication * In the Multitudes app, go to [Settings > Integrations](https://app.multitudes.co/team-settings/integrations) and click on the “Connect” button in the Deployments API card.
* This will take you to the API Keys tab. Click “Generate New Key”.
* In the modal, create an Organization key that has the Deployments write scope
* After the key has been generated, copy it an use as the Bearer token your API calls (see further instructions below). The full key will only be shown once
## Configure Multitudes to use deployments from the API The [Deployment Frequency](/metrics-and-definitions/process-metrics/value-delivery/deployment-frequency) chart shows deployments from either GitHub Actions or the deployments you send via the API. To configure which source Multitudes uses, you can set the toggle in the [Organization Settings](https://app.multitudes.co/team-settings/organization-settings?section=configure-deployments) page under the Configure deployments section.
## POST a deployment To tell us about a deployment, make an HTTP POST request to our endpoint with the generated authentication token in the header and required parameters. You should make one POST per repository being deployed and include the commit SHAs of the merge commit for each PR being deployed. This is particularly important if you are deploying multiple PRs at once or if your PRs are being merged via an intermediary branch. Note that we include each deployment we receive a POST for in our metric calculations. We recommend that you only POST deployments to your production environment; that will match your metrics to the DORA approach, which focuses on deploys to production. See more about [how we calculate Deployment Frequency](/metrics-and-definitions/process-metrics/value-delivery/deployment-frequency). ### **Path** ``` https://integrations.multitudes.co/deployment ``` ### **Body parameters** * `commitSha` *string or string\[]* **Required** The SHA-1 value(s) of the commit(s) being deployed. Each value should have a minimum of 7 characters. While this is likely fine for most organizations, you may wish to send us a longer SHA if you’d like more assurance that they will be matched to the correct commit and PR data ([see here for more information on the uniqueness of short SHA-1 values](https://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#Short-SHA-1)). Commits should all be from the same repository. This will determine how the our repositories filter interacts with charts like [Deployment Frequency](/metrics-and-definitions/process-metrics/value-delivery/deployment-frequency). If there are commits from different repos in the `commit_sha` field, we will assume the repository of the first commit in the list. Commits that are authored by users who are not a [Contributor](/configuration-and-setup/permissions-and-roles) in Multitudes will be filtered out of results. If a deployment has no commits authored by Multitudes contributors, the whole deployment will be filtered out of results as we won't be able to attribute it to a Multitudes contributor nor their team. If a deployment only has commits authored by bots, these will be included in the grey Organization line only. * `environmentName` *string* **Required***‍* The environment that this is deploying to. This field appears in the [Deployment Frequency](/metrics-and-definitions/process-metrics/value-delivery/deployment-frequency) drilldown table to help you identify individual deployments. This is particularly useful in a monorepo situation where you might have deployments from one repo going to multiple production environments - you're free to put whatever is most useful for you in there. {% hint style="warning" %} **IMPORTANT** Only POST deployments when you consider something deployed to production. Posting development deployments will cause Change Lead Time to be miscalculated. Anything you send will be displayed and included in metrics as we don't filter out non-production environments. {% endhint %} If you send us deployments to multiple environments for the same commits, we will take the time of the latest deploy to calculate the `Deploy Time` (and therefore `Change Lead Time`) of a matched PR. This is because we assume the PR hasn't been fully released until it has been deployed to all environments. If you send us deployments with the same commits and environment name, we will dedupe by taking the latest received deployment. * `deployedAt` *UTC datetime in ISO8601 format* The time of the deployment.\ *Default: the timestamp that the POST request was received by our API* * `title` *string* *‍*A title to give your deployment for easy identification on our drilldown tables.\ *Default: the commit message of the matched commit with the most recent commit timestamp* * `tags` *string* A custom field for you to add labels (e.g., \["is\_fix", "is\_failure", "sev1", "product\_launch"]). In the future, we plan to support filtering on these labels.\ *Default: none* ### **Sending a Test deployment** When you set up the Deployments API, you can send us a Test Deployment to ensure its working. To do so, add the following Body parameter: \ `isTest` *boolean true/false* Any deployments that include the `isTest` parameter will be excluded from any data analysis. \ *Default: false* ### **Responses** Below is formatted as `code`, `response text`, description * `201`, `Created successfully` , Deploy registered successfully. * `400`, `Invalid request` , The data is not in the expected format, the response will provide a list of fields that have an issue. * `401`, `Unauthorized` , API key not valid. * `403`, `Forbidden`, API key does not have the correct scopes or organisation for this request or has been revoked. * `500`, `Internal server error`. Please contact for more information. An unknown error has occurred, please contact support with the returned request ID for us to investigate. ## **Examples** ### **curl** `--fail-with-body` flag to ensure cURL will exit with code 22 on an HTTP failure status code (>=400). Otherwise, you can look for a non-20x status code to detect a failure. ```sh curl --request POST \ --fail-with-body \ --url "https://integrations.multitudes.co/deployment" \ --header "Content-Type: application/json" \ --header "Authorization: $MULTITUDES_API_KEY" \ --data '{"commitSha": "$COMMIT_SHA", "environmentName":"$ENVIRONMENT"}' ``` With a list of commit SHAs: ```sh curl --request POST \ --fail-with-body \ --url "https://integrations.multitudes.co/deployment" \ --header "Content-Type: application/json" \ --header "Authorization: $MULTITUDES_API_KEY" \ --data '{ "commitSha": [ "8750d79cce5f2d137c3f4a34cdd4c9da76c26cfb", "561b41361698b579e40f0f9a68491a0e64f48849" ], "environmentName": "test" }' ``` Sending a test deployment: ```sh curl --request POST \ --fail-with-body \ --url "https://integrations.multitudes.co/deployment" \ --header "Content-Type: application/json" \ --header "Authorization: $MULTITUDES_API_KEY" \ --data '{ "commitSha": "8750d79cce5f2d137c3f4a34cdd4c9da76c26cfb", "environmentName": "test", "isTest": true }' ``` ## **Retrieving deployment data** After sending a deployment through the API, you can verify the transmitted data using our GET endpoint with the generated authentication token in the header. ### **Path** ``` https://integrations.multitudes.co/deployment ``` ### **Query parameters** * `page` *number* The page number of of the deployments.\ *Default: 1*
* `pageSize` *number* The number of deployments to show per page. Maximum is 100.\ *Default: 20*
* `includeTests` *boolean true/false* Whether to include deployments with `isTest` set to `true` in the response.\ *Default: false*\ \ *E.g. pageSize 20, page 2, would return deployments 21-40.* ### **Responses** Below is formatted as `code`, `response text`, description * `200`, `OK` , Retrieved deploys successfully. * `401`, `Unauthorized` , API key not valid. * `403`, `Forbidden`, API key does not have the correct scopes or organisation for this request or has been revoked. * `500`, `Internal server error`. Please contact for more information. An unknown error has occurred, please contact support for us to investigate. ## **Examples** ### **curl** `--fail-with-body` flag to ensure cURL will exit with code 22 on an HTTP failure status code (>=400). Otherwise, you can look for a non-20x status code to detect a failure. ```sh curl --request GET \ --fail-with-body \ --url "https://integrations.multitudes.co/deployment" \ --header "Content-Type: application/json" \ --header "Authorization: $MULTITUDES_API_KEY" ``` Including test deployments: ```sh curl --request GET \ --fail-with-body \ --url "https://integrations.multitudes.co/deployment?includeTests=true" \ --header "Content-Type: application/json" \ --header "Authorization: $MULTITUDES_API_KEY" ``` With different page and page size: ```sh curl --request GET \ --fail-with-body \ --url "https://integrations.multitudes.co/deployment?page=2&pageSize=25" \ --header "Content-Type: application/json" \ --header "Authorization: $MULTITUDES_API_KEY" ``` ## GitHub Actions Example with a single commit SHA: ```sh jobs: log-deployment: runs-on: ubuntu-latest needs: [ ] environment: ${{ inputs.environment }} steps: - run: | curl --request POST \ --fail-with-body \ --url "hhttps://integrations.multitudes.co/deployment" \ --header "Content-Type: application/json" \ --header "Authorization: ${{ secrets.MULTITUDES_API_KEY }}" \ --data '{"commitSha": "${{github.sha}}","environmentName": "${{inputs.environment}}"}' ``` Example collecting all SHAs since the last commit to this branch (*this is especially useful if your feature branches are merged to a `develop`/`staging`/`test` branch and then that branch is eventually merged into a "production" branch for deployment*): ```sh jobs: log-deployment: runs-on: ubuntu-latest needs: [ ] environment: ${{ inputs.environment }} steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - run: | PREV_DEPLOY=HEAD^1 CURRENT_DEPLOY=HEAD # get the commit sha's of all commits after the previous deploy to the current deploy REV_LIST=`git rev-list --ancestry-path $PREV_DEPLOY..$CURRENT_DEPLOY` # make the rev_list into json REV_LIST=[\"${REV_LIST//[^a-f0-9]/\", \"}\"] # post the deploy details curl --request POST \ --fail-with-body \ --url "https://integrations.multitudes.co/deployment" \ --header "Content-Type: application/json" \ --header "Authorization: ${{ secrets.MULTITUDES_API_KEY }}" \ --data "{\"commitSha\": $REV_LIST, \"environmentName\": \"${{inputs.environment}}\"}" ``` Learn more about Deployments Metrics [here](/metrics-and-definitions/deployment-metrics) # GitHub Actions How to connect GitHub Actions to Multitudes ## GitHub Actions + Multitudes Integration Benefits * See [`Deploy Time`](/metrics-and-definitions/process-metrics/flow-of-work/deploy-time) - how long code takes to get deployed once a PR has been merged and include Deploy Time in [`Change Lead Time`](/metrics-and-definitions/process-metrics/flow-of-work/change-lead-time) calculations * See how often code is successfully deployed to Production with [`Deployment Frequency`](/metrics-and-definitions/process-metrics/value-delivery/deployment-frequency) ([a key DORA metric](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance)) * Keep track of [`Deployment Failure Rate`](/metrics-and-definitions/process-metrics/quality-of-work/deployment-failure-rate) {% hint style="info" %} You can use *either* our [Deployments API ](/integrations/deployments-api)*or* our GitHub Actions integration to provide us with data about deployments. If you already have our Deployments API integration configured, you will need to revoke that action token before connecting with our GitHub Actions integration. {% endhint %} ## How it works [See here for details](https://docs.multitudes.com/metrics-and-definitions/deployment-metrics#if-youre-using-our-github-actions-integration) on how GitHub Actions data is used to calculate Deployment metrics in Multitudes. ## Requirements Any GitHub organisation admin can set up and configure your GitHub Actions integration. ## How to install 1. In the Multitudes app, go to [Settings > Integrations](https://app.multitudes.co/team-settings/integrations) and click on the Connect button in the Github and Github Actions card
2. On the resulting modal, click to configure Github Actions data
3. On the next page of the modal, you can choose either the “Environment/Deployments” or “Workflows” option, based on how you use GitHub Actions. See details below the image.
Don't know what to pick? Here are some common scenarios! **“The change is deployed when it has been deployed to a certain environment/s"** * This sounds like you are using GitHub Action’s “Environments” feature, and therefore their “Deployments” feature, so you would select "Environment/Deployments" * In your case, the best way for us to track a “deployment” is for you to tell us which Environment/s are your production environments in each repository. You'll see this in the next step. * Note that within one workflow run, you can have more than 1 successful deployment (e.g., you could use the same workflow to deploy to dev, staging, and prod). Therefore it’s important to understand how we [calculate deployment metrics](/metrics-and-definitions/deployment-metrics) when you’ve selected this option. {% hint style="info" %} **A note on historical data availability:** Due to constraints on GitHub’s API, we will not be able to fetch historical deployment data for this option. * This is only an issue if you onboarded to Multitudes recently (i.e., if you onboarded to Multitudes a while ago but didn't configure GitHub Actions until today, then today we can still access your historical deployment data). * While most charts will have 6 weeks of historical data from before you onboarded onto Multitudes to get you started, charts relating to deployments will only have data from onboarding onwards. * Also note that it might look like your `Change Lead Time` increases from the date that you onboard, because from that point onwards, change lead time will include deploy time based on your deployments data. * Here's a concrete example * You onboard onto Multitudes (install our GitHub app) on 1 June, from this point on we start accumulating real-time events data * We also pull 6 weeks of historical data, back to 16 May, to get you started * On 8 June, you configure GitHub Actions * We will only show deployment data from June 1 to June 8 {% endhint %} **“The change is deployed when a specific workflow(s) has run”** * This sounds like you are not using GitHub Action’s “Environments” feature * In your case, the best way for us to track a “deployment” is for you to tell us which Workflow/s are the ones that deploy to your production environment for each repository. You'll see this in the next step. {% hint style="info" %} **A note on historical data availability:** we will have access to historical data (the 6 weeks of data from before you onboarded onto Multitudes), but only the latest attempt in the historical data. This means that for example, if an [attempt to deploy to production](/metrics-and-definitions/deployment-metrics) succeeded at 12pm, but for some reason you re-ran the workflow again successfully at 1pm, 1pm will be what is captured as the Deploy Time for that PR, even though it actually got deployed earlier at 12pm. This is a constraint on the historical data from the GitHub API. {% endhint %} With respect to data availability in either case, once you have integrated GitHub Actions, we will have real-time data, and will take the first successful attempts to deploy to production going forward (since that is when the change is first available to customers). 4. Once you’ve made a selection, the following page of the modal will prompt you for further configuration. In either case you will: * First identify which repository * Then within that repository, select either relevant environment or workflow * Click the "Save" button Screenshot for if you selected “Environment/Deployments
Screenshot for if you selected “Workflows
5. All done!
# Google Calendar Install our Google Calendar integration to get insights like Focus Time and Meeting Load. ## Google Calendar + Multitudes Integration Benefits * See how much [`Focus Time`](/metrics-and-definitions/process-metrics/flow-of-work/focus-time) your team is getting * Keep track of [`Meeting Load`](/metrics-and-definitions/people-metrics/wellbeing/meeting-load) – the number of hours spent in meetings per team and user, during both work hours and out-of-hours (customisable by each team member's usual working hours) ## How it works Calendar events are sent from Google Calendar to Multitudes. This data is then processed by our data pipelines and shown in the Multitudes app in our Flow of Work and Wellbeing insights. ## Requirements In order to install this integration, you must have both: * Owner or Manager level [permissions](/configuration-and-setup/permissions-and-roles) on Multitudes, and * Google Workspace Admin access to set up the connection to your organization’s Google Workspace {% hint style="warning" %} **Note:** The Google Workspace integration will allow you to proceed with integration if you are not a Google Workspace Admin. However, this will not provide Multitudes a sufficient view of calendar data for our analyses. If you have installed as a non-admin please contact to support with the reinstall. {% endhint %} ## **How to install** On the Multitudes app, go to the [Integrations page](https://app.multitudes.co/team-settings/integrations) (it’s in Settings > Integrations). 1. Find the card that says Google Calendar and click Connect in the top right of the card.
2. After you click Connect, on the resulting pop-up click Connect Google.
3. Choose the Google account you want to link with Multitudes. You must be a Google Workplace admin to complete the following steps.
4. Click Continue.
5. This will open up permission settings. Click Allow.
**Why does Multitudes need these permissions?**‍ * **See info about users on your domain:** We use this to automatically link up users from the Google Workspace with Multitudes Contributors. * **View events on all your calendars:** We use this to build the [Focus Time](/metrics-and-definitions/process-metrics/flow-of-work/focus-time) and [Meeting Load](/metrics-and-definitions/people-metrics/wellbeing/meeting-load) charts * **Edit events on all your calendars:** We don’t currently allow people to edit any calendar events from within Multitudes. However, we have this scope turned on because in future iterations we will give users the ability to update and edit their calendars from the 1:1 section of Multitudes.
6. You will be brought back into the application. Next, we’ll ask you to link Google users to Multitudes users. Where the email address in Google matches what we have in Multitudes, we will automatically link those users (example below). If the users don’t get picked up automatically (i.e. if the email address of the Multitudes Contributor is different from their Google email), you can manually link a Multitudes Contributor to a Google User account by choosing the appropriate Google account from the drop-down menu.
7. Then click continue and you’re done! {% hint style="info" %} Note: Private events are included in Multitudes metrics but meeting titles and attendees are redacted. If you would like to redact **all** meeting details, contact and we will enable this setting for your organisation. {% endhint %} ## Reconnecting Google Calendar to a new account If you find that you want to reconnect Google Calendar to Multitudes using a different Google account, follow this link here: [app.multitudes.co/google/connect](http://app.multitudes.co/google/connect) # Outlook Calendar Install our Outlook Calendar integration to get insights like Focus Time and Meeting Load. ## Outlook Calendar + Multitudes Integration Benefits * See how much [`Focus Time`](/metrics-and-definitions/process-metrics/flow-of-work/focus-time) your team is getting * Keep track of [`Meeting Load`](/metrics-and-definitions/people-metrics/wellbeing/meeting-load) – the number of hours spent in meetings per team and user, during both work hours and out-of-hours (customisable by each team member's usual working hours) ## How it works Calendar events are sent from Outlook Calendar to Multitudes. This data is then processed by our data pipelines and shown in the Multitudes app in our Flow of Work and Wellbeing insights. ## Requirements Note that in order to install this integration, you must have both: * Owner or Manager level [permissions](/configuration-and-setup/permissions-and-roles) on Multitudes, and * Privileged Role Administrator access to set up the connection to your organization’s Outlook app {% hint style="info" %} **Note:** If you are not an admin you will be prompted to ask an admin for approval during the integration setup through Microsoft 365. {% endhint %} ## **How to install** On the Multitudes app, go to the [Integrations page](https://app.multitudes.co/team-settings/integrations) (it’s in Settings > Integrations). 1. Find the card that says Outlook Calendar and click Connect in the top right of the card.
Screen in Multitudes app showing integration tiles including Outlook Calendar tile and Connect button.
2. After you click Connect, on the resulting pop-up click Connect Outlook.
Modal to connect Multitudes to Outlook Calendar - says we connect directly to Microsoft 365 and the connection requires a Cloud Application Administrator.
3. Choose the Microsoft account you want to link with Multitudes.
Microsoft screen to pick an account
4. This will open up permission settings. Review and click Accept.
Microsoft screen outlines the Multitudes app would like to read and write calendars in all mailboxes, read profile photo of a user, read all users' basic profiles and sign in and read user profile.
**Why does Multitudes need these permissions?**‍ * **Read and write calendars in all mailboxes:** We use this to build the [Focus Time](/metrics-and-definitions/process-metrics/flow-of-work/focus-time) and [Meeting Load](/metrics-and-definitions/people-metrics/wellbeing/meeting-load) charts * **Read profile photo of a user or group, Read all users' basic profiles, Sign in and read user profile:** We use these permissions to access the user information we need to link up users from Outlook with Multitudes Contributors. 6. You will be brought back into the application. Next, we’ll ask you to link Outlook users to Multitudes users. Where the email address in Outlook matches what we have in Multitudes, we will automatically link those users (example below). If the users don’t get picked up automatically (i.e. if the email address of the Multitudes Contributor is different from their Outlook email), you can manually link a Multitudes Contributor to an Outlook User account by choosing the appropriate Outlook account from the drop-down menu.
Screen with list of Multitudes contributors next to dropdown fields to link to Outlook users.
7. Then click continue and you’re done! {% hint style="info" %} Note: Private events are included in Multitudes metrics but meeting titles and attendees are redacted. If you would like to redact **all** meeting details, contact and we will enable this setting for your organisation. {% endhint %} # Jira Install our Jira integration to get insights like Types of Work and Feature vs Maintenance Work. ## Jira + Multitudes Integration Benefits * Get visibility over velocity and spread of work with [`Types of Work`](/metrics-and-definitions/process-metrics/value-delivery/types-of-work) insights * See the balance between [`Feature and Maintenance work` ](/metrics-and-definitions/process-metrics/value-delivery/feature-vs-maintenance-work) ## How it works Issue data from Jira will be sent to Multitudes. This data is then processed by our data pipelines and shown in the Multitudes app as actionable insights around [`Value Delivery`](/metrics-and-definitions/process-metrics/value-delivery). We pull up to 200 Jira projects, ordered by the latest issue updated. We check for new projects and update this list daily. We pull up to 200 epics and 200 labels. The Jira API does not provide us a way to sort these or filter by any sort of date field, so they are just the first 200 returned by their API. Your Jira insights can be categorised based on Issue Type, Epic, and/or Label (see [Customize Work Categories](https://docs.multitudes.com/configuration-and-setup/customize-work-categories#how-to-customize-work-completed)). ## Requirements In order to install this integration, you must have both: * Owner or Manager level [permissions](/configuration-and-setup/permissions-and-roles) on Multitudes, and * Jira System Administrator role to set up the connection to your organization’s Jira account ## How to install On the Multitudes app, go to the [Integrations page](https://app.multitudes.co/team-settings/integrations) (from the menu, find Account, click Settings, then click the Integrations tab across the top). Find the card that says Jira in the top Integrations section and click ‘Connect’ at top right. 1. After you click 'Connect' on the Jira card on the Integrations page, on the resulting pop-up click ‘Connect Jira projects’
2. This opens a new page on Atlassian Marketplace, click ‘Try it free’
3. This pops up a modal. In this modal, first, select your site; second, click the button to ‘Start free trial’. Don’t worry this integration is not separately charged beyond your instance of Multitudes!
4. This pops up a new modal, click ‘Get started’
5. This will close the modals, on the resulting page, a dialog box should pop up on the lower left hand corner indicating that the install is in progress. Once complete, it will show success, from this dialog box, click ‘Get started’
6. A new tab will open, on this tab, click ‘Allow access’
7. A new tab will open, scroll to the bottom and click ‘Accept’
8. This will close the last tab. On the resulting page, click the button ‘Connect with Multitudes’
9. You'll be redirected back to our app where you’ll see a pop-up asking you to link Jira users with Multitudes users. This will determine whose Jira data we know to show in Multitudes, so it's important to complete this step. [Learn more about user linking](/configuration-and-setup/user-linking). {% hint style="info" %} When populating the user linking list, we include the names of any users who appear as the Creator, Assignee, or Reporter for any issue in our database. {% endhint %} 10. Lastly, you’ll see a success message in the pop-up!
{% hint style="warning" %} Once the installation is finished and teams are linked, you will need to **wait a few minutes, then refresh the app** to see the charts. Until the charts are ready, you will see the message "No data for this time period" on the Value Delivery page (screenshot below). {% endhint %} ![Screenshot](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/6458455fdd37f07c6a5280a6_9305440b.png) ![Screenshot](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/645939f0ea5b2a627edc1234_Screenshot%202023-05-04%20at%209.14.29%20PM.png) ## How to Uninstall Follow the link below – you’ll need to input your Atlassian org: ``` https://[YOUR_ATLASSIAN_ORG].atlassian.net/plugins/servlet/upm ``` Then find the listing for Multitudes for Jira and click “uninstall”.
# Linear Install our Linear integration to get insights like Types of Work and Feature vs Maintenance Work. ## Linear + Multitudes Integration Benefits * Get visibility over velocity and spread of work with [`Types of Work`](/metrics-and-definitions/process-metrics/value-delivery/types-of-work) insights * See the balance between [`Feature and Maintenance work` ](/metrics-and-definitions/process-metrics/value-delivery/feature-vs-maintenance-work) ## How it works Issue data from Linear will be sent to Multitudes. This data is then processed by our data pipelines and shown in the Multitudes app as actionable insights around [`Value Delivery`](/metrics-and-definitions/process-metrics/value-delivery). The data can be viewed by either Issue count or Story Points, depending on your ways of working, and categorised based on labels or projects (see [Customize Work Categories](https://docs.multitudes.com/configuration-and-setup/customize-work-categories#how-to-customize-work-completed)). ## Requirements In order to install this integration, you must have both: * Owner or Manager level [permissions](/configuration-and-setup/permissions-and-roles) on Multitudes, and * Admin role in Linear ## How to install On the Multitudes app, go to the [Integrations page](https://app.multitudes.co/team-settings/integrations) (from the menu, find Account, click Settings, then click the Integrations tab across the top). Find the card that says Linear in the top Integrations section and click ‘Connect’ at top right. 1. After you click 'Connect' on the Linear card on the Integrations page, on the resulting pop-up click ‘Continue to set up Linear’
2. This opens a new page, click 'Authorize Multitudes'
3. Now, you should be brought back to our app, where you’ll see a pop-up asking you to link Linear users with Multitudes users. This will determine whose Linear data we know to show in Multitudes, so it's important to complete this step. [Learn more about user linking](/configuration-and-setup/user-linking). 4. Lastly, you’ll see a success message in the pop-up!
## Linear integration stopped working? Due to Linear's authentication system, the integration will stop working if the team member who originally set up the connection leaves your organization. If your integration stops working, you can force the reinstall by following this link: # Opsgenie How to connect Opsgenie to Multitudes ## OpsGenie + Multitudes Integration Benefits * See [`Mean Time To Recovery`](/metrics-and-definitions/process-metrics/quality-of-work/mean-time-to-recovery) for OpsGenie incidents ## How it works Incident-related events from OpsGenie (such as an incident being triggered, assigned, acknowledged, and resolved) will be sent to Multitudes. This data is then processed by our data pipelines and shown in the Multitudes app. ## Requirements In order to install this integration, you must have both: * Owner or Manager level [permissions](/configuration-and-setup/permissions-and-roles) on Multitudes, and * Account owner or Global Admin access to generate the API key in OpsGenie ## How to install * In the Multitudes app, go to [Settings > Integrations](https://app.multitudes.co/team-settings/integrations) and click on the connect Opsgenie card. This should direct you to your Opsgenie instance (otherwise click [here](https://app.opsgenie.com/alert/list)) * On Opsgenie navigate to settings > API key management > add a new API key
* Once you click add a new API key, you should get a pop-up. Proceed to do 3 things in the pop-up
1. Select both the "Read" and “Configuration access” access scopes 2. Copy the key - we also recommend you store this API access key in a secure location 3. Click "Add API Key" to finalize * Back on Multitudes, paste the previously-copied API key into the text field and click save * You will be prompted to link Opsgenie users with Multitudes users. This will determine whose Opsgenie data we know to show in Multitudes, so it's important to complete this step. [Learn more about user linking](/configuration-and-setup/user-linking). * All done! Multitudes will now start to track your incidents and calculate `Mean Time to Restore` {% hint style="warning" %} Multitudes tracks MTTR by looking at your Opsgenie `incidents`. It does **not** look at Opsgenie `alerts`. The Opsgenie `incidents` feature is only available to paying Opsgenie customers. {% endhint %} # PagerDuty How to connect PagerDuty to Multitudes ## PagerDuty + Multitudes Integration Benefits * See [`Mean Time To Recovery`](/metrics-and-definitions/process-metrics/quality-of-work/mean-time-to-recovery) for PagerDuty incidents * See [`Mean Time To Acknowledge`](/metrics-and-definitions/process-metrics/quality-of-work/mean-time-to-acknowledge) for PagerDuty incidents * See [`Number of Pages`](/metrics-and-definitions/process-metrics/quality-of-work/number-of-pages) split by service and escalation policy * Keep track of burnout risk by seeing [`Page Disruptions`](/metrics-and-definitions/people-metrics/wellbeing/page-disruptions) – by either the number of pages or number of disrupted hours per team and user, during both work hours and out-of-hours (customisable by each team member's usual working hours) ## How it works Incident-related events from PagerDuty (such as an incident being triggered, assigned, acknowledged, and resolved) will be sent to Multitudes. This data is then processed by our data pipelines and shown in the Multitudes app as actionable insights around [`Quality of Work`](/metrics-and-definitions/process-metrics/quality-of-work) and [`Wellbeing`](/metrics-and-definitions/people-metrics/wellbeing). ## Requirements PagerDuty integrations require an Admin base role for account authorization. If you do not have this role, please reach out to a PagerDuty Admin or Account Owner within your organization to configure the integration. If you need help with this integration, please contact `support@multitudes.com`. ## How to install 1. Go to 2. Find the **PagerDuty** card and click **Connect**. 3. In the modal, click **Continue to set up PagerDuty**. This will open the PagerDuty sign in page. 4. Sign in with your PagerDuty credentials. 5. **Submit Consent** for the app installation. This will redirect you back to Multitudes.You will see a new modal screen that prompts you to input an API key.
Screenshot of the approval screen on PagerDuty

Step 5: approve the pemissions by clicking "Submit Consent"

6. For the next step, go to PagerDuty in a new tab. 7. In the top navigation bar, navigate to **Integrations** > **API Access Keys**. 8. Click **Create new API key.**‍ 9. Add a description, e.g. "Multitudes App Integration", do **not** select "Read-only API Key"\*, and click **Create Key**.\ \*We need write access to create the webhook on your PagerDuty account.
10. Copy the newly generated **API Key**, then **Close**. 11. Back in the Multitudes tab, paste the API Key in the text field on the modal, and click **Add API Key**.
A screenshot of the Multitudes PagerDuty configuration modal, showing where to input the API key.
This is the installation part complete. You can now revoke the API token in PagerDuty as we only use it to set up the webhook. In the top navigation bar, navigate to **Integrations > API Access Keys**, find the API key you just created, and click **Remove** on the right hand side of the row. The next two steps in Multitudes help you configure your data so that it is accurate and comprehensive. 1. User linking: see the [section on user linking](/configuration-and-setup/user-linking) for how to match your Multitudes team members to PagerDuty users, so that incident assignments and pages are attributed to the correct people in Multitudes. 2. Default filter setting: choose which field would be most useful for you to filter incidents on, between **urgency** and **priority level.** ## **How we use PagerDuty's webhooks**
Screenshot of PagerDuty showing the organization's list of webhooks from the Integrations tab.

List of webhooks on PagerDuty

We use your API token to create a webhook. You can see this in PagerDuty by navigating to **Integrations** > **Generic Webhooks (v3)**. The webhook sends us events on updates to incidents, which powers our PagerDuty-related insights about incidents and pages in Multitudes.
Screenshot of PagerDuty showing the "Manage Webhook" screen for a Multitudes webhook.

Webhook settings shown when you click into the webhook.

{% hint style="danger" %} Please don't edit this webhook, as it will impact the delivery of data into Multitudes and affect the accuracy of your insights. This includes the `multitudesId` custom header is used to identify your PagerDuty organization inside Multitudes. {% endhint %} If you have any questions about the settings, please contact `support@multitudes.com`. ## Historic Data Once the data from Pagerduty has been ingested, you will see that the `Mean Time To Recovery` (MTTR) graph will be populated with the last six weeks of data. Due to limitations from the Pagerduty API, the `Mean Time To Acknowledge` (MTTA) will not display historic data and will instead be populated with data from the point at which the webhook was created. ## How to Uninstall To uninstall the app, please contact `support@multitudes.com`. To uninstall the webhook: 1. In PagerDuty, in the top navigation bar, navigate to **Integrations** > **Generic Webhooks (v3)**. 2. Under **Subscriptions**, find the webhook called `https://pagerduty.prod.multitudes.co/events`. Click the **"..." menu** on the right hand side of the card, and click **Delete**. # Slack How to connect Slack notifications to Multitudes ## Slack + Multitudes Integration Benefits Multitudes can proactively send you notifications to help you improve performance, collaboration, and wellbeing, in your existing workflow. [View available Slack notifications. ](/knowledge-base/types-of-notifications) ## How it works We have both team and personal Slack Notifications. Once Multitudes has been enabled in your organization's Slack workspace, teams can set up their own channel notifications (e.g. ad hoc [PR updates](/knowledge-base/types-of-notifications/pr-updates), the daily [Work Digest](/knowledge-base/types-of-notifications/work-digest), and more) and individuals can enable personal notifications. ## Requirements Different organizations have different permissions required to install apps to a Slack workspace. If your workspace requires "[app approval](https://slack.com/intl/en-nz/help/articles/222386767-Manage-app-approval-for-your-workspace)" you will need a Slack Workspace Owner to install or approve the Multitudes integration. ## **How to install** Choose whether you need to set up Slack for your organization or for you as an individual – jump to the relevant instructions: * [Slack organization setup](#slack-organization-setup) – to receive team notifications in shared channels * [Slack individual setup](#for-personal-pr-updates) – to receive Multitudes notifications to yourself via DM Note that you can also set up notifications [from the My Insights page](#from-the-my-insights-page) – instructions below. ### **From the** [**Settings > Notifications**](https://app.multitudes.co/team-settings/notifications) **page** In the Multitudes app, click your user icon at the bottom of the navigation sidebar. Click Settings, then go to the "Notifications" tab. 1. Click one of the "Connect" buttons. 2. In the modal that pops up, click the “Install the Multitudes for Slack app” button. This will open a new window on your browser, at `[workspace-name].slack.com`.\ ‍**Make sure you’re installing on the correct workspace – you can change this in the dropdown on the top right corner.** 3. Click “Allow” (or “Request approval” if your Slack organization has been set up to require approval from a workspace owner). 4. **To get alerts in private channels**, you need to add the `Multitudes for Slack` app to those specific channels. You can do this by typing `@multitudes` in each private channel, or clicking on the private channel's name and going to `Integrations > Add apps` . #### **Slack organization setup** 1. Go to [**Settings > Notifications > Team**](https://app.multitudes.co/team-settings/notifications?section=team)**.** Then click on **Connect your Slack organization**. 1. This will open a new window on your browser, at `[workspace-name].slack.com`.
2. Go through the Slack installation modal. 1. Make sure you’re installing on the correct workspace - you can change this in the dropdown on the top right corner. 2. Review the app permissions needed by Multitudes to access Slack. 3. Click “Allow” (or “Request approval” if your Slack organization has been set up to require approval from a workspace owner).
**Note: To get notifications in private channels**, you need to add the `Multitudes for Slack` app to those specific channels. You can do this by typing `@multitudes` in each private channel, or clicking on the private channel's name and going to `Integrations > Add apps`. #### Slack individual setup 1. Go to [**Settings > Notifications > Personal**](https://app.multitudes.co/team-settings/notifications?section=personal). Then click on **Connect your Slack DMs**.
2\. Review the app permissions needed by Multitudes to access Slack. Then click **Allow**.
### **From the** [My Insights](https://app.multitudes.co/) **page** For convenience, you can also connect to Slack and configure [team notifications](/configuration-and-setup/notifications-configuration) directly from the My Insights homepage by clicking the bell icons. ![Screenshot](https://cdn.prod.website-files.com/610c8a14b4df1ae46b1a13a3/64caa792888c778fe0fbe501_homepage%20for%20alerts.png) {% hint style="info" %} Learn more about how to configure your notifications [here](/configuration-and-setup/notifications-configuration). {% endhint %} ## Explore our Slack Notifications
PR Updates – ad hoc/pages/odB5zJD1A6dozBEdVjGz
Work Digest – daily/pages/izWYNFa7lVE7O6aekHRf
Trend Summary – weekly/pages/olXjslxCTaD0S8iYg2e4
1:1 Prompts/pages/DiwQ5SsvM51AP5rJXUeP
Annotations/pages/TwThXenFIHXqP69ObqPl
# Claude Code How to connect Claude Code to Multitudes to get AI adoption and impact metrics ## Claude Code + Multitudes Integration Benefits Track Claude Code usage and impact across your engineering team with insights on adoption patterns, usage trends, and productivity metrics. See more: * [AI Adoption](/metrics-and-definitions/ai-impact-feature/ai-adoption) * [AI Impact](/metrics-and-definitions/ai-impact-feature/ai-impact) ## **What’s your Claude Code plan?** * **API or Enterprise** – read on below. * **Team or Individual** – use our Open Telemetry option to send us data. Docs here: [Send Open Telemetry data to Multitudes](https://docs.multitudes.com/integrations/open-telemetry) See more about the Claude Code plans here: ## Claude Code: How it works Claude Code usage data is sent to Multitudes via an **Admin (for the API plan) or an Analytics (for the Enterprise plan) API key**. This data is processed by our data pipelines and shown in the Multitudes app as actionable insights around AI adoption and engineering productivity. {% hint style="warning" %} The latest Claude Code data can take up to **3 days** before they appear in the app. This is a current limitation of their API, where data is only available for querying after 3 days. For example, the data for January 1, 2026 will only be available for querying on January 3, 2026. {% endhint %} ## How to install #### From the [Settings > Integrations](https://app.multitudes.co/team-settings/integrations) page 1. Find the Claude Code card and click **Connect**. 2. In the modal, you'll be prompted to select your plan and input an API key.
3. Follow the instructions to generate an API key for the plan you're on below: 1. [API plan](#api-plan-instructions) 2. [Enterprise plan](#enterprise-plan-instructions) 4. Copy the generated API key and paste it in the text field. 5. Click **Connect Claude Code**. ### API plan instructions #### Requirements **IMPORTANT:** This Claude Code integration option requires your organization to be on an [**API plan**](https://claude.com/pricing#api) and have access to the [Claude Console](https://platform.claude.com/dashboard). This Claude Code integration requires an **Admin API key** (not a standard API key) for us to receive the correct data. If you do not have access to generate **Admin API keys**, please reach out to an administrator within your organization to configure the integration. If you need help with this integration, please contact . #### Generate your Claude Code Admin API key You can do this [directly here](https://platform.claude.com/settings/admin-keys). 1. Go to your Claude Code admin settings. 2. Navigate to the API keys section. 3. Click **Create new Admin API key** (note: standard API keys will not work). 4. Add a description, e.g. "Multitudes Integration". 5. Copy the newly generated Admin API key.
### Enterprise plan instructions #### Requirements **IMPORTANT:** This Claude Code integration option requires your organization to be on an [**Enterprise plan**](https://claude.com/pricing#team-&-enterprise). This Claude Code integration requires an **Analytics API key** (not a standard API key) for us to receive the correct data. If you do not have access to generate **Analytics API keys**, please reach out to an administrator within your organization to configure the integration. You'll need to enable access to the Analytics API and create a key with the `read:analytics` scope. If you need help with this integration, please contact . #### Generate your Claude Code Analytics API key You can do this [directly here](http://claude.ai/analytics/api-keys). 1. Go to your Claude Code Analytics page. 2. Navigate to the API keys section. 3. Enable access if it isn't already enabled. 4. Click **Create key** (note: standard API keys will not work). 5. Enable the `read:analytics` scope. 6. Add a description, e.g. "Multitudes Integration". 7. Copy the newly generated Analytics API key.
## User linking Complete the user linking step to match your Multitudes team members to Claude Code users, ensuring usage data is attributed to the correct people in Multitudes.
## Historic data Once the data from Claude Code has been ingested, you will see usage metrics populated with the last 12 weeks of data. ## How to uninstall To uninstall the integration, please contact . # Open Telemetry (OTel) We currently accept Open Telemetry metrics from Claude Code and Codex. Other tools coming soon. ## What is OTel? OTel is an open-source, vendor-neutral standard for generating, collecting, and exporting telemetry data such as metrics, logs, and traces. Multitudes uses OpenTelemetry to receive your AI tooling usage data. For more on OpenTelemetry, see the [official documentation](https://opentelemetry.io/docs/what-is-opentelemetry/?utm_source=chatgpt.com). ## How it works ### What metrics we can ingest and process We can ingest and process OTel data for: * Claude Code [Team](https://claude.com/pricing/team) and [Individual](https://claude.com/pricing) plans * Codex If you're on the Claude Code [API](https://claude.com/pricing#api) or [Enterprise](https://claude.com/pricing/enterprise) plan, head to our docs here: [Claude Code](https://docs.multitudes.com/integrations/claude-code). {% hint style="info" %} If you'd like to bring in Open Telemetry data from other AI tools besides Claude Code or Codex, let us know at . {% endhint %} ### Integration options We can ingest OTel metrics in two different ways: #### 1. Multitudes-hosted collector Claude Code and Codex users send metrics directly to the Multitudes Open Telemetry collector running in Multitudes infrastructure. This avoids having to host a collector in your own infrastructure. To do this, follow steps here: [Multitudes-hosted collector](#multitudes-hosted-collector). Or jump to specific instructions for: * [Claude Code](#for-claude-code) * [Codex](#for-codex) #### 2. Self-hosted collector You either run your own collector or the Multitudes Open Telemetry collector in your infrastructure. The collector ingests metrics from Claude Code and Codex users and forwards them to Multitudes. To do this, follow the steps below: [Self-hosted collector](#self-hosted-collector) ## Multitudes-hosted collector When using the Multitudes-hosted collector, each user sends their telemetry data directly to the Multitudes endpoint. For this, each user needs a configuration file that specifies the endpoint to use and an `Authorization` header. Apply the following configuration to each user's machine: #### For Claude Code: 1. Open the `~/.claude/settings.json` . 2. Add the following lines, using an API key that has been generated (following the instructions below): ```json "env": { "CLAUDE_CODE_ENABLE_TELEMETRY": "1", "OTEL_METRICS_EXPORTER": "otlp", "OTEL_LOGS_EXPORTER": "otlp", "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf", "OTEL_EXPORTER_OTLP_ENDPOINT": "https://integrations.multitudes.co/otel", "OTEL_METRIC_EXPORT_INTERVAL": "10000", "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer " }, ``` {% hint style="warning" %} We’re starting new work to measure AI effectiveness. If you’d like to help us out and be beta testers for these upcoming features, include the following in your `~/.claude/settings.json`: **Tool usage logging** ``` "OTEL_LOG_TOOL_DETAILS": "1" ``` This turns on detailed logging of tool usage in telemetry traces, which includes Bash commands, MCP server and tool names, skill names, prompt length, and the input arguments passed to each tool. This redacts the contents of prompts unless you have specifically enabled that (see next section). **Bonus: Prompt logging** ``` "OTEL_LOG_USER_PROMPTS": "1" ``` If you like, you can also turn this on to send us the contents of prompts that your organization is using. We do not receive the contents of any attached files unless they are explicitly included in the prompt text. More about these environment variables in the [Claude Code docs](https://code.claude.com/docs/en/monitoring-usage). Note that enabling both of these settings is optional and will not change anything in the Multitudes app now. But enabling this data will give us real user data to test new features on and you’ll gain early access to these new features as we build them. {% endhint %} {% hint style="info" %} The `Authorization` header can be an Organization API key if you are provisioning this file via MDM and want to use the same key across multiple users. If you would prefer that each user has their own key, they can create a Personal API key and update their `~/.claude/settings.json` file. {% endhint %} {% hint style="info" %} **AWS Bedrock:** when using Bedrock the `user.email` attribute must be configured manually via the `OTEL_RESOURCE_ATTRIBUTES` environment variable. This is because Bedrock authentication does not automatically provide user email information. Each user will need to add their work email address (or whichever email is associated with their user in Multitudes): ```json "env": { ... "OTEL_RESOURCE_ATTRIBUTES": "user.email=developer@company.com" } ``` {% endhint %} #### For Codex: 1. Open the `~/.codex/config.toml`. 2. Add the following lines: ```toml [otel] exporter = { otlp-http = { endpoint = "https://integrations.multitudes.co/otel/v1/logs", protocol = "binary", headers = { "Authorization" = "Bearer " } } } log_user_prompt = false ``` 3. Each user should [generate a personal API key](/configuration-and-setup/api-keys) 4. Replace `` with your actual API key. Restart any sessions you might have in Claude. {% hint style="info" %} **Authenticated via API Key:** If logging in or authenticating Codex using an API key (regardless if it's the desktop app or the CLI), each team member will also need to add the following to their shell profile (e.g. `~/.zshrc` or `~/.bashrc`), since API key auth doesn't carry account identity the way ChatGPT login does: ```bash export OTEL_RESOURCE_ATTRIBUTES="user.email=developer@company.com" ``` {% endhint %} ## Self-hosted collector When using the self-hosted collector, each user sends their telemetry data to a collector run in your infrastructure, from which aggregated metrics are sent to Multitudes. ### Running the collector 1. Set up the Multitudes OTel Collector. Detailed instructions can be found [here](https://github.com/MultitudesCo/otel-collector). 2. [Follow the instructions to generate an Organization API key](/configuration-and-setup/api-keys) and run the collector using this key. ### Sending from an existing collector 1. If you have an existing collector, update your config to add an exporter that sends to the Multitudes ingestion endpoint. The main sections to configure are in `exporters` , `processors`, and `service`. 2. Add a new `otlphttp/multitudes` exporter ```yaml exporters: otlphttp/multitudes: endpoint: https://integrations.multitudes.co/otel encoding: json headers: Authorization: "Bearer ${env:MULTITUDES_API_KEY}" timeout: 30s retry_on_failure: enabled: true initial_interval: 5s max_interval: 30s max_elapsed_time: 300s ``` 3. Add the new `otlphttp/multitudes` to your existing metrics and logs pipelines: ```yaml service: pipelines: metrics: receivers: [...your existing receivers] processors: [...your existing processors] exporters: [...your existing exporters, otlphttp/multitudes] logs: receivers: [...your existing receivers] processors: [...your existing processors] exporters: [...your existing exporters, otlphttp/multitudes] ``` 4. Follow the instructions below to generate an Organization API key and run the collector using this key. ### Sending data to the collector Apply the following configuration to each user's machine: ```json { "env": { "CLAUDE_CODE_ENABLE_TELEMETRY": "1", "OTEL_METRICS_EXPORTER": "otlp", "OTEL_LOGS_EXPORTER": "otlp", "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf", "OTEL_EXPORTER_OTLP_ENDPOINT": "", "OTEL_METRIC_EXPORT_INTERVAL": "10000" } } ``` {% hint style="info" %} **AWS Bedrock:** when using Bedrock the `user.email` attribute must be configured manually via the `OTEL_RESOURCE_ATTRIBUTES` environment variable. This is because Bedrock authentication does not automatically provide user email information. Each user will need to add their work email address (or whichever email is associated with their user in Multitudes): ```json "env": { ... "OTEL_RESOURCE_ATTRIBUTES": "user.email=developer@company.com" } ``` {% endhint %} ## Checking data ingestion status Once you are sending data to Multitudes, either directly or via the self-hosted option, you can check the ingestion status in the Multitudes app, Open the [OpenTelemetry integration modal](https://app.multitudes.co/team-settings/integrations?pill=true\&integration=otel) to check whether your data is coming through.
Once data is being received successfully, your Claude usage metrics will appear in Multitudes after our batch processing pipeline runs. Our pipeline runs daily, so in most cases you should expect to see metrics populate in the Multitudes App within up to one day of sending data. # Cursor How to connect Cursor to Multitudes to get AI adoption and impact metrics ### Cursor + Multitudes Integration Benefits Track Cursor usage and impact across your engineering team with insights on adoption patterns, usage trends, and productivity metrics. See more: * [AI Adoption](/metrics-and-definitions/ai-impact-feature/ai-adoption) * [AI Impact](/metrics-and-definitions/ai-impact-feature/ai-impact) ### How it works Cursor usage data is sent to Multitudes via an API key. This data is processed by our data pipelines and shown in the Multitudes app as actionable insights around AI adoption and engineering productivity. ### Requirements Cursor integrations require an "Admin API key" for us to receive the correct data. If you do not have access to generate API keys, please reach out to an administrator within your organization to configure the integration. If you need help with this integration, please contact . ### How to install #### From the [Settings > Integrations](https://app.multitudes.co/team-settings/integrations) page 1. Go to 2. Find the Cursor card and click **Connect**. 3. In the modal, you'll be prompted to input an API key (this will need to be an "Admin API Key")
#### Generate your Cursor API key 1. Open the [Cursor dashboard](https://cursor.com/dashboard) in a browser 2. Open the Settings tab 3. Navigate to the **Advanced** section (see screenshot below) 4. Click **New Admin API Key** 5. Add a description, e.g. "Multitudes Integration" 6. Copy the newly generated API key **Note:** Members can generate API keys in Cursor, and while they have the admin scope, they can only be used for requests relating to that specific member – so these keys don't provide the analytics we need for this integration. If you can see the Settings>Advanced page below, then you're an admin. If you can't see the page below, then you'll need to ask a Cursor admin to generate the key.
#### Complete the integration 1. Return to the Multitudes tab 2. Paste the API key in the text field on the modal 3. Click **Add API Key** #### User linking Complete the user linking step to match your Multitudes team members to Cursor users, ensuring usage data is attributed to the correct people in Multitudes. ### Historic Data Once the data from Cursor has been ingested, you will see usage metrics populated with the last 12 weeks of data. ### How to Uninstall To uninstall the integration, please contact . # Github Copilot How to connect GitHub Copilot to Multitudes to get AI adoption and impact metrics ### GitHub Copilot + Multitudes Integration Benefits Track GitHub Copilot usage and impact across your engineering team with insights on adoption patterns, usage trends, acceptance rates, and productivity metrics. See more: * [AI Adoption](/metrics-and-definitions/ai-impact-feature/ai-adoption) * [AI Impact](/metrics-and-definitions/ai-impact-feature/ai-impact) ### How it works GitHub Copilot usage data is retrieved from the GitHub's Copilot Metrics API and Copilot Usage Metrics API. This data is processed by our data pipelines and shown in the Multitudes app as actionable insights around AI adoption and AI impact. ### Requirements GitHub Copilot integrations require: * GitHub Enterprise Cloud or GitHub Team Edition **with Copilot enabled** * **Organization admin permissions to enable Copilot Metrics API access** * Acceptance of new Multitudes GitHub App permission requests for existing GitHub integration customers If you need help with this integration, please contact . ### How to configure You don't need to integrate with a new tool as we already natively integrate with Github. #### For existing GitHub integration customers 1. Ensure you've accepted the latest permission requests from the Multitudes GitHub App 2. Within your GitHub organization settings, enable the **Copilot Metrics API** and **Copilot Usage Metrics** access #### Enable Copilot Metrics in GitHub 1. Go to your GitHub enterprise or organization copilot settings 1. For GitHub Enterprise: Navigate to [`github.com/enterprises/{organization_name}/ai-controls/copilot`](http://github.com/enterprises/{organization_name}/ai-controls/copilot) 2. If you don’t have Enterprise, then you’ll set it at the organization level: Navigate to [`github.com/organizations/{organization_name}/settings/copilot/policies`](http://github.com/organizations/{organization_name}/settings/copilot/policies) 2. Enable **Copilot Metrics API** access 3. Enable **Copilot Usage Metrics** access
{% hint style="info" %} **Inclusion in Github Copilot Metrics** 1. For the endpoints to retrieve Copilot data to work, the Copilot Metrics API access policy must be enabled for the organization. 2. In order for an end user to be counted towards these metrics, they must have telemetry enabled in their IDE. {% endhint %} #### Verify the Integration on Multitudes 1. In the [Settings > Integrations](https://app.multitudes.co/team-settings/integrations) page, find the GitHub Copilot card and click **Connect**. 2. In the modal, click **Verify Copilot integration**.
3. If you've accepted all of the necessary permissions, then the modal should close and a pop-up should appear at the top-right saying that you've successfully verified Copilot.
# Multitudes API Export data from Multitudes via API The Multitudes API is a public REST API that lets you interact with the Multitudes app programmatically. You can query chart data, retrieve drilldown data for all metrics in the app, and retrieve metadata about your organization. All data is returned in a consistent JSON format that makes it easy to combine with other data sources or build custom workflows around your engineering data. ## Before you start All endpoints are served from `https://api.multitudes.co`. An OpenAPI spec is available without authentication at [`https://api.multitudes.co/v1/openapi.json`](https://api.multitudes.co/v1/openapi.json). Authenticate using an organization API key passed as `Authorization: Bearer `. Keys must have the `data:read` scope to access metric endpoints. See how to generate an API key [here](/configuration-and-setup/api-keys). ## Rate limits Current rate limits are as follows: * 6 requests per second * 30 requests per minute * 150 requests per hour These limits are per key and are set to ensure API and app performance with the API load we're expecting. We'll be monitoring API usage and revisiting these rate limits over time. ## Versioning All endpoints are versioned, with the current version being v1. ## Endpoints ## GET /v1/teams > List all teams in the organization. ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"servers":[{"url":"/customer"}],"security":[{"bearerAuth":[]}],"components":{"securitySchemes":{"bearerAuth":{"scheme":"bearer","bearerFormat":"JWT","type":"http","description":"Authorization: Bearer . A raw token in the Authorization header is also accepted for backward compatibility."}},"schemas":{"TeamsResponseDto":{"type":"object","properties":{"items":{"description":"List of teams.","type":"array","items":{"$ref":"#/components/schemas/TeamDto"}}},"required":["items"]},"TeamDto":{"type":"object","properties":{"id":{"type":"string","description":"Team UUID."},"name":{"type":"string","description":"Team display name."},"status":{"type":"string","description":"Team status.","enum":["active","archived"]}},"required":["id","name","status"]}}},"paths":{"/v1/teams":{"get":{"parameters":[{"name":"includeArchived","required":false,"in":"query","description":"When true, archived teams are included in the response.","schema":{"default":false,"type":"boolean"}}],"responses":{"200":{"description":"","content":{"application/json":{"schema":{"$ref":"#/components/schemas/TeamsResponseDto"}}}}},"summary":"List all teams in the organization.","tags":["customer"]}}}} ``` ## List all supported metric types. > Returns each metric with a description and the optional query parameters it accepts beyond the standard ones (from, to, timescale, teamIds, teamNames). ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"servers":[{"url":"/customer"}],"security":[{"bearerAuth":[]}],"components":{"securitySchemes":{"bearerAuth":{"scheme":"bearer","bearerFormat":"JWT","type":"http","description":"Authorization: Bearer . A raw token in the Authorization header is also accepted for backward compatibility."}},"schemas":{"MetricsListResponseDto":{"type":"object","properties":{"items":{"type":"array","items":{"$ref":"#/components/schemas/MetricInfoDto"}}},"required":["items"]},"MetricInfoDto":{"type":"object","properties":{"type":{"type":"string","description":"Metric type identifier.","enum":["leadTime","codingTime","reviewWaitTime","editingTime","deployTime","mergeFrequency","changeFailureRate","linesOfCodeChanged","filesChanged","deploymentFailureRate","commitsOutOfHours","deploymentFrequency","feedbackGiven","feedbackReceived","feedbackParticipationGap","githubFeatureVsMaintenance","aiActiveUsers","aiIntensityOfUsage","aiCohortDistribution","focusTimeGoogle","focusTimeOutlook","meetingHoursGoogle","meetingHoursOutlook","meetingHoursOohGoogle","meetingHoursOohOutlook","feedbackQuality","feedbackThemes","pagerdutyMeanTimeToRestore","pagerdutyMeanTimeToAcknowledge","pagerdutyPageDisruptions","pagerdutyPageDisruptionsOoh","pagerdutyNumberOfPages","jiraTypeOfWork","jiraFeatureVsMaintenance","linearTypeOfWork","linearFeatureVsMaintenance"]},"description":{"type":"string","description":"Human-readable description of what this metric measures."},"acceptedQueryParams":{"description":"Optional query parameters supported by this metric, beyond the standard ones (from, to, timescale, teamIds, teamNames).","type":"array","items":{"type":"string"}},"drilldown":{"type":"boolean","description":"Whether GET /customer/v1/charts/:metricType/drilldown is supported for this metric."}},"required":["type","description","acceptedQueryParams","drilldown"]}}},"paths":{"/v1/metrics":{"get":{"description":"Returns each metric with a description and the optional query parameters it accepts beyond the standard ones (from, to, timescale, teamIds, teamNames).","parameters":[],"responses":{"200":{"description":"","content":{"application/json":{"schema":{"$ref":"#/components/schemas/MetricsListResponseDto"}}}}},"summary":"List all supported metric types.","tags":["customer"]}}}} ``` ## Retrieve chart data for a metric. > Returns time-series or aggregate chart data for the specified metric. Use GET /customer/v1/metrics to discover supported metrics and their accepted query parameters. ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"servers":[{"url":"/customer"}],"security":[{"bearerAuth":[]}],"components":{"securitySchemes":{"bearerAuth":{"scheme":"bearer","bearerFormat":"JWT","type":"http","description":"Authorization: Bearer . A raw token in the Authorization header is also accepted for backward compatibility."}},"schemas":{"ChartResponseDto":{"type":"object","properties":{"metricType":{"type":"string","description":"Metric type identifier."},"from":{"type":"string","description":"Query start date as ISO 8601 timestamp. Absent for aggregate view."},"to":{"type":"string","description":"Query end date as ISO 8601 timestamp. Absent for aggregate view."},"timescale":{"type":"string","description":"Timescale used for data point bucketing. Absent for aggregate view.","enum":["daily","weekly","monthly"]},"options":{"description":"Echoes options that were explicitly provided in the request.","allOf":[{"$ref":"#/components/schemas/ChartOptionsDto"}]},"series":{"description":"Chart series.","type":"array","items":{"$ref":"#/components/schemas/SeriesDto"}},"units":{"description":"Unit of measurement for this metric's values. A plain string for all metrics except aiIntensityOfUsage, which returns an object keyed by value field (spend, inputTokens, linesChanged) since that metric emits three parallel units per data point.","oneOf":[{"type":"string"},{"type":"object","additionalProperties":{"type":"string"}}]}},"required":["metricType","options","series","units"]},"ChartOptionsDto":{"type":"object","properties":{"repositories":{"description":"Repository names the response was filtered to.","type":"array","items":{"type":"string"}},"excludeOrganization":{"type":"boolean","description":"Whether the org-level rollup series was excluded from the response."},"excludeWeekendHours":{"type":"boolean","description":"Whether weekend hours were excluded."},"includeSelfiePrs":{"type":"boolean","description":"Whether self-review PRs were included."},"defaultBranchOnly":{"type":"boolean","description":"Whether only default-branch commits/PRs were included."},"view":{"type":"string","description":"View type for githubFeatureVsMaintenance.","enum":["timeSeries","aggregate"]},"direction":{"type":"string","description":"Feedback direction. Applies to feedbackQuality and feedbackThemes metrics.","enum":["given","received"]},"groupBy":{"type":"string","description":"How to group chart series. Defaults to team. AI metrics support: jobLevel, timezone, aiTool, nonContributors. PagerDuty metrics support: service, escalationPolicy.","enum":["team","service","escalationPolicy","jobLevel","timezone","aiTool","nonContributors"]},"cohortMetric":{"type":"string","description":"Cohort metric used to classify users (aiCohortDistribution only).","enum":["dailyActiveUsage","spend","inputTokens","linesChanged"]},"cohortThreshold":{"type":"number","description":"Threshold value used to split High/Low AI cohorts (aiCohortDistribution only). Unit depends on cohortMetric: percentage of active days for 'dailyActiveUsage', USD cents for 'spend', token count for 'inputTokens', line count for 'linesChanged'."},"cohortLookbackWeeks":{"type":"number","description":"Number of lookback weeks used for cohort classification (aiCohortDistribution only)."},"individual":{"type":"boolean","description":"When true, return one series per team member instead of per team. Supported for meeting-hours calendar, PR cycle, and GitHub code review metrics only.","default":false},"countBy":{"type":"string","description":"Unit to count by. For issue tracker metrics (jiraTypeOfWork, jiraFeatureVsMaintenance, linearTypeOfWork, linearFeatureVsMaintenance): issues (default) or storyPoints. For GitHub feedback metrics (feedbackGiven, feedbackReceived, feedbackParticipationGap): comments (default) or reviews. For page disruption metrics (pagerdutyPageDisruptions, pagerdutyPageDisruptionsOoh): pages (default) or disruptedHours.","enum":["issues","storyPoints","comments","reviews","pages","disruptedHours"]},"priorities":{"description":"PagerDuty priority IDs the response was filtered to (PagerDuty metrics only).","type":"array","items":{"type":"string"}},"urgencies":{"type":"array","description":"Urgency levels the response was filtered to (PagerDuty metrics only).","items":{"type":"string","enum":["high","low"]}}}},"SeriesDto":{"type":"object","properties":{"id":{"type":"string","description":"Series identifier. For team series: team UUID. For cohort series: cohort name."},"name":{"type":"string","description":"Display name for the series."},"type":{"type":"string","description":"Series type.","enum":["team","organization","member","cohort"]},"dataPoints":{"description":"Ordered list of data points for this series.","type":"array","items":{"$ref":"#/components/schemas/DataPointDto"}}},"required":["id","name","type","dataPoints"]},"DataPointDto":{"type":"object","properties":{"startDate":{"type":"string","description":"Bucket start date (ISO 8601)."},"endDate":{"type":"string","description":"Bucket end date (ISO 8601)."},"value":{"type":"number","description":"Metric value. Omitted for comment metrics (commentsGiven, commentsReceived, commentsParticipationRatio) and when percentile data (p50/p75/p90/p95) is present."},"p50":{"type":"number","description":"50th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p75":{"type":"number","description":"75th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p90":{"type":"number","description":"90th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p95":{"type":"number","description":"95th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"count":{"type":"number","description":"Number of data points considered in this bucket (e.g. PRs, commits, issues). Null indicates no data for this bucket.","nullable":true},"memberCount":{"type":"number","description":"Number of people considered in this bucket (team members in scope for the data point). Null indicates no data for this bucket.","nullable":true},"spend":{"type":"number","description":"Total AI spend in USD cents for this bucket. Present for aiIntensityOfUsage.","nullable":true},"inputTokens":{"type":"number","description":"Total input tokens consumed for this bucket. Present for aiIntensityOfUsage.","nullable":true},"linesChanged":{"type":"number","description":"Total lines of code accepted for this bucket. Present for aiIntensityOfUsage.","nullable":true},"total":{"type":"number","description":"Total (sum) value for this bucket. Null indicates no data for this bucket.","nullable":true},"label":{"type":"string","description":"Category label for aggregate view data points (e.g. 'Feature', 'Bug'). Only present for githubFeatureVsMaintenance and leadTime when view=aggregate."}},"required":["startDate","endDate"]}}},"paths":{"/v1/charts/{metricType}":{"get":{"description":"Returns time-series or aggregate chart data for the specified metric. Use GET /customer/v1/metrics to discover supported metrics and their accepted query parameters.","parameters":[{"name":"from","required":true,"in":"query","description":"Start of the date range as an ISO 8601 datetime. Required.","schema":{"type":"string"}},{"name":"to","required":true,"in":"query","description":"End of the date range as an ISO 8601 datetime. Required.","schema":{"type":"string"}},{"name":"timescale","required":false,"in":"query","description":"Timescale for bucketing data points.","schema":{"default":"weekly","enum":["daily","weekly","monthly"],"type":"string"}},{"name":"teamIds","required":false,"in":"query","description":"Comma-separated team IDs to filter (mutually exclusive with teamNames, max 50).","schema":{"type":"string"}},{"name":"teamNames","required":false,"in":"query","description":"Comma-separated team names to filter (mutually exclusive with teamIds, max 50).","schema":{"type":"string"}},{"name":"repositories","required":false,"in":"query","description":"Comma-separated repository names to filter (GitHub metrics only).","schema":{"type":"string"}},{"name":"excludeWeekendHours","required":false,"in":"query","description":"Exclude weekend hours from calculations. Applies to PR cycle time metrics.","schema":{"default":false,"type":"boolean"}},{"name":"includeSelfiePrs","required":false,"in":"query","description":"Include PRs where the author also reviewed. Default false.","schema":{"default":false,"type":"boolean"}},{"name":"defaultBranchOnly","required":false,"in":"query","description":"Limit to commits/PRs on default branch only. Applies to githubFeatureVsMaintenance.","schema":{"default":false,"type":"boolean"}},{"name":"excludeOrganization","required":false,"in":"query","description":"Exclude the org-level rollup series from the response. Default false.","schema":{"default":false,"type":"boolean"}},{"name":"view","required":false,"in":"query","description":"View type for githubFeatureVsMaintenance. Default timeSeries.","schema":{"enum":["timeSeries","aggregate"],"type":"string"}},{"name":"groupBy","required":false,"in":"query","description":"How to group chart series. Defaults to team. AI metrics support: jobLevel, timezone, aiTool, nonContributors. PagerDuty metrics support: service, escalationPolicy.","schema":{"default":"team","enum":["team","service","escalationPolicy","jobLevel","timezone","aiTool","nonContributors"],"type":"string"}},{"name":"individual","required":false,"in":"query","description":"When true, return one series per team member instead of per team. Supported for meeting-hours calendar, PR cycle, and GitHub code review metrics only.","schema":{"default":false,"type":"boolean"}},{"name":"countBy","required":false,"in":"query","description":"Unit to count by. For issue tracker metrics (jiraTypeOfWork, jiraFeatureVsMaintenance, linearTypeOfWork, linearFeatureVsMaintenance): issues (default) or storyPoints. For GitHub feedback metrics (feedbackGiven, feedbackReceived, feedbackParticipationGap): comments (default) or reviews. For page disruption metrics (pagerdutyPageDisruptions, pagerdutyPageDisruptionsOoh): pages (default) or disruptedHours.","schema":{"enum":["issues","storyPoints","comments","reviews","pages","disruptedHours"],"type":"string"}},{"name":"direction","required":false,"in":"query","description":"Direction for feedback quality/themes metrics: 'given' (comments the team wrote) or 'received' (comments written on the team's PRs). Default given.","schema":{"default":"given","enum":["given","received"],"type":"string"}},{"name":"priorities","required":false,"in":"query","description":"Comma-separated PagerDuty priority IDs to filter by (PagerDuty metrics only). Use the priority IDs from your PagerDuty configuration.","schema":{"type":"string"}},{"name":"urgencies","required":false,"in":"query","description":"Comma-separated urgency levels to filter by (PagerDuty metrics only). Valid values: high, low.","schema":{"type":"string"}},{"name":"metricType","required":true,"in":"path","description":"The metric to retrieve.","schema":{"enum":["leadTime","codingTime","reviewWaitTime","editingTime","deployTime","mergeFrequency","changeFailureRate","linesOfCodeChanged","filesChanged","deploymentFailureRate","commitsOutOfHours","deploymentFrequency","feedbackGiven","feedbackReceived","feedbackParticipationGap","githubFeatureVsMaintenance","aiActiveUsers","aiIntensityOfUsage","aiCohortDistribution","focusTimeGoogle","focusTimeOutlook","meetingHoursGoogle","meetingHoursOutlook","meetingHoursOohGoogle","meetingHoursOohOutlook","feedbackQuality","feedbackThemes","pagerdutyMeanTimeToRestore","pagerdutyMeanTimeToAcknowledge","pagerdutyPageDisruptions","pagerdutyPageDisruptionsOoh","pagerdutyNumberOfPages","jiraTypeOfWork","jiraFeatureVsMaintenance","linearTypeOfWork","linearFeatureVsMaintenance"],"type":"string"}}],"responses":{"200":{"description":"","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ChartResponseDto"}}}},"400":{"description":"Invalid query parameters."},"404":{"description":"Metric type not found or not available for your organization."}},"summary":"Retrieve chart data for a metric.","tags":["customer"]}}}} ``` {% hint style="info" %} Note that the unit of `value` and percentile fields (`p50`, `p75`, `p90`, `p95`) is "hours" for the following metrics: `leadTime`, `codingTime`, `reviewWaitTime`, `editingTime`, and `deployTime` {% endhint %} ## List AI super-users. > Returns the top AI coding tool users over the last 28 days. ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"servers":[{"url":"/customer"}],"security":[{"bearerAuth":[]}],"components":{"securitySchemes":{"bearerAuth":{"scheme":"bearer","bearerFormat":"JWT","type":"http","description":"Authorization: Bearer . A raw token in the Authorization header is also accepted for backward compatibility."}},"schemas":{}},"paths":{"/v1/super-users":{"get":{"description":"Returns the top AI coding tool users over the last 28 days.","parameters":[{"name":"teamIds","required":false,"in":"query","description":"Comma-separated team IDs to filter (mutually exclusive with teamNames, max 50).","schema":{"type":"string"}},{"name":"teamNames","required":false,"in":"query","description":"Comma-separated team names to filter (mutually exclusive with teamIds, max 50).","schema":{"type":"string"}}],"responses":{"200":{"description":"Super-user list.","content":{"application/json":{"schema":{"properties":{"items":{"type":"array","items":{"$ref":"#/components/schemas/PublicSuperUserDto"}}}}}}}},"summary":"List AI super-users.","tags":["customer"]}}}} ``` ## Models ## The PostDeploymentDto object ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"components":{"schemas":{"PostDeploymentDto":{"type":"object","properties":{"commitSha":{"description":"Commit SHA(s) for the deployment. Can be a single commit SHA string or an array of commit SHA strings.","oneOf":[{"type":"string","pattern":"^[A-Fa-f0-9]{7,40}$"},{"type":"array","minItems":1,"maxItems":1000,"items":{"type":"string","pattern":"^[A-Fa-f0-9]{7,40}$"}}]},"environmentName":{"type":"string","description":"The environment name the deployment occurred in.","minLength":1},"deployedAt":{"type":"string","description":"When the deployment happened (ISO8601 date-time).","format":"date-time"},"title":{"type":"string","description":"Optional title/label for the deployment.","minLength":1},"tags":{"description":"Optional tags for the deployment.","maxItems":1000,"items":{"type":"string"},"type":"array"},"isTest":{"type":"boolean","description":"If true, the deployment is treated as a test deployment and may be handled differently."}},"required":["commitSha","environmentName"]}}}} ``` ## The TeamDto object ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"components":{"schemas":{"TeamDto":{"type":"object","properties":{"id":{"type":"string","description":"Team UUID."},"name":{"type":"string","description":"Team display name."},"status":{"type":"string","description":"Team status.","enum":["active","archived"]}},"required":["id","name","status"]}}}} ``` ## The TeamsResponseDto object ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"components":{"schemas":{"TeamsResponseDto":{"type":"object","properties":{"items":{"description":"List of teams.","type":"array","items":{"$ref":"#/components/schemas/TeamDto"}}},"required":["items"]},"TeamDto":{"type":"object","properties":{"id":{"type":"string","description":"Team UUID."},"name":{"type":"string","description":"Team display name."},"status":{"type":"string","description":"Team status.","enum":["active","archived"]}},"required":["id","name","status"]}}}} ``` ## The MetricInfoDto object ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"components":{"schemas":{"MetricInfoDto":{"type":"object","properties":{"type":{"type":"string","description":"Metric type identifier.","enum":["leadTime","codingTime","reviewWaitTime","editingTime","deployTime","mergeFrequency","changeFailureRate","linesOfCodeChanged","filesChanged","deploymentFailureRate","commitsOutOfHours","deploymentFrequency","feedbackGiven","feedbackReceived","feedbackParticipationGap","githubFeatureVsMaintenance","aiActiveUsers","aiIntensityOfUsage","aiCohortDistribution","focusTimeGoogle","focusTimeOutlook","meetingHoursGoogle","meetingHoursOutlook","meetingHoursOohGoogle","meetingHoursOohOutlook","feedbackQuality","feedbackThemes","pagerdutyMeanTimeToRestore","pagerdutyMeanTimeToAcknowledge","pagerdutyPageDisruptions","pagerdutyPageDisruptionsOoh","pagerdutyNumberOfPages","jiraTypeOfWork","jiraFeatureVsMaintenance","linearTypeOfWork","linearFeatureVsMaintenance"]},"description":{"type":"string","description":"Human-readable description of what this metric measures."},"acceptedQueryParams":{"description":"Optional query parameters supported by this metric, beyond the standard ones (from, to, timescale, teamIds, teamNames).","type":"array","items":{"type":"string"}},"drilldown":{"type":"boolean","description":"Whether GET /customer/v1/charts/:metricType/drilldown is supported for this metric."}},"required":["type","description","acceptedQueryParams","drilldown"]}}}} ``` ## The MetricsListResponseDto object ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"components":{"schemas":{"MetricsListResponseDto":{"type":"object","properties":{"items":{"type":"array","items":{"$ref":"#/components/schemas/MetricInfoDto"}}},"required":["items"]},"MetricInfoDto":{"type":"object","properties":{"type":{"type":"string","description":"Metric type identifier.","enum":["leadTime","codingTime","reviewWaitTime","editingTime","deployTime","mergeFrequency","changeFailureRate","linesOfCodeChanged","filesChanged","deploymentFailureRate","commitsOutOfHours","deploymentFrequency","feedbackGiven","feedbackReceived","feedbackParticipationGap","githubFeatureVsMaintenance","aiActiveUsers","aiIntensityOfUsage","aiCohortDistribution","focusTimeGoogle","focusTimeOutlook","meetingHoursGoogle","meetingHoursOutlook","meetingHoursOohGoogle","meetingHoursOohOutlook","feedbackQuality","feedbackThemes","pagerdutyMeanTimeToRestore","pagerdutyMeanTimeToAcknowledge","pagerdutyPageDisruptions","pagerdutyPageDisruptionsOoh","pagerdutyNumberOfPages","jiraTypeOfWork","jiraFeatureVsMaintenance","linearTypeOfWork","linearFeatureVsMaintenance"]},"description":{"type":"string","description":"Human-readable description of what this metric measures."},"acceptedQueryParams":{"description":"Optional query parameters supported by this metric, beyond the standard ones (from, to, timescale, teamIds, teamNames).","type":"array","items":{"type":"string"}},"drilldown":{"type":"boolean","description":"Whether GET /customer/v1/charts/:metricType/drilldown is supported for this metric."}},"required":["type","description","acceptedQueryParams","drilldown"]}}}} ``` ## The ChartOptionsDto object ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"components":{"schemas":{"ChartOptionsDto":{"type":"object","properties":{"repositories":{"description":"Repository names the response was filtered to.","type":"array","items":{"type":"string"}},"excludeOrganization":{"type":"boolean","description":"Whether the org-level rollup series was excluded from the response."},"excludeWeekendHours":{"type":"boolean","description":"Whether weekend hours were excluded."},"includeSelfiePrs":{"type":"boolean","description":"Whether self-review PRs were included."},"defaultBranchOnly":{"type":"boolean","description":"Whether only default-branch commits/PRs were included."},"view":{"type":"string","description":"View type for githubFeatureVsMaintenance.","enum":["timeSeries","aggregate"]},"direction":{"type":"string","description":"Feedback direction. Applies to feedbackQuality and feedbackThemes metrics.","enum":["given","received"]},"groupBy":{"type":"string","description":"How to group chart series. Defaults to team. AI metrics support: jobLevel, timezone, aiTool, nonContributors. PagerDuty metrics support: service, escalationPolicy.","enum":["team","service","escalationPolicy","jobLevel","timezone","aiTool","nonContributors"]},"cohortMetric":{"type":"string","description":"Cohort metric used to classify users (aiCohortDistribution only).","enum":["dailyActiveUsage","spend","inputTokens","linesChanged"]},"cohortThreshold":{"type":"number","description":"Threshold value used to split High/Low AI cohorts (aiCohortDistribution only). Unit depends on cohortMetric: percentage of active days for 'dailyActiveUsage', USD cents for 'spend', token count for 'inputTokens', line count for 'linesChanged'."},"cohortLookbackWeeks":{"type":"number","description":"Number of lookback weeks used for cohort classification (aiCohortDistribution only)."},"individual":{"type":"boolean","description":"When true, return one series per team member instead of per team. Supported for meeting-hours calendar, PR cycle, and GitHub code review metrics only.","default":false},"countBy":{"type":"string","description":"Unit to count by. For issue tracker metrics (jiraTypeOfWork, jiraFeatureVsMaintenance, linearTypeOfWork, linearFeatureVsMaintenance): issues (default) or storyPoints. For GitHub feedback metrics (feedbackGiven, feedbackReceived, feedbackParticipationGap): comments (default) or reviews. For page disruption metrics (pagerdutyPageDisruptions, pagerdutyPageDisruptionsOoh): pages (default) or disruptedHours.","enum":["issues","storyPoints","comments","reviews","pages","disruptedHours"]},"priorities":{"description":"PagerDuty priority IDs the response was filtered to (PagerDuty metrics only).","type":"array","items":{"type":"string"}},"urgencies":{"type":"array","description":"Urgency levels the response was filtered to (PagerDuty metrics only).","items":{"type":"string","enum":["high","low"]}}}}}}} ``` ## The DataPointDto object ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"components":{"schemas":{"DataPointDto":{"type":"object","properties":{"startDate":{"type":"string","description":"Bucket start date (ISO 8601)."},"endDate":{"type":"string","description":"Bucket end date (ISO 8601)."},"value":{"type":"number","description":"Metric value. Omitted for comment metrics (commentsGiven, commentsReceived, commentsParticipationRatio) and when percentile data (p50/p75/p90/p95) is present."},"p50":{"type":"number","description":"50th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p75":{"type":"number","description":"75th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p90":{"type":"number","description":"90th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p95":{"type":"number","description":"95th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"count":{"type":"number","description":"Number of data points considered in this bucket (e.g. PRs, commits, issues). Null indicates no data for this bucket.","nullable":true},"memberCount":{"type":"number","description":"Number of people considered in this bucket (team members in scope for the data point). Null indicates no data for this bucket.","nullable":true},"spend":{"type":"number","description":"Total AI spend in USD cents for this bucket. Present for aiIntensityOfUsage.","nullable":true},"inputTokens":{"type":"number","description":"Total input tokens consumed for this bucket. Present for aiIntensityOfUsage.","nullable":true},"linesChanged":{"type":"number","description":"Total lines of code accepted for this bucket. Present for aiIntensityOfUsage.","nullable":true},"total":{"type":"number","description":"Total (sum) value for this bucket. Null indicates no data for this bucket.","nullable":true},"label":{"type":"string","description":"Category label for aggregate view data points (e.g. 'Feature', 'Bug'). Only present for githubFeatureVsMaintenance and leadTime when view=aggregate."}},"required":["startDate","endDate"]}}}} ``` ## The SeriesDto object ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"components":{"schemas":{"SeriesDto":{"type":"object","properties":{"id":{"type":"string","description":"Series identifier. For team series: team UUID. For cohort series: cohort name."},"name":{"type":"string","description":"Display name for the series."},"type":{"type":"string","description":"Series type.","enum":["team","organization","member","cohort"]},"dataPoints":{"description":"Ordered list of data points for this series.","type":"array","items":{"$ref":"#/components/schemas/DataPointDto"}}},"required":["id","name","type","dataPoints"]},"DataPointDto":{"type":"object","properties":{"startDate":{"type":"string","description":"Bucket start date (ISO 8601)."},"endDate":{"type":"string","description":"Bucket end date (ISO 8601)."},"value":{"type":"number","description":"Metric value. Omitted for comment metrics (commentsGiven, commentsReceived, commentsParticipationRatio) and when percentile data (p50/p75/p90/p95) is present."},"p50":{"type":"number","description":"50th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p75":{"type":"number","description":"75th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p90":{"type":"number","description":"90th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p95":{"type":"number","description":"95th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"count":{"type":"number","description":"Number of data points considered in this bucket (e.g. PRs, commits, issues). Null indicates no data for this bucket.","nullable":true},"memberCount":{"type":"number","description":"Number of people considered in this bucket (team members in scope for the data point). Null indicates no data for this bucket.","nullable":true},"spend":{"type":"number","description":"Total AI spend in USD cents for this bucket. Present for aiIntensityOfUsage.","nullable":true},"inputTokens":{"type":"number","description":"Total input tokens consumed for this bucket. Present for aiIntensityOfUsage.","nullable":true},"linesChanged":{"type":"number","description":"Total lines of code accepted for this bucket. Present for aiIntensityOfUsage.","nullable":true},"total":{"type":"number","description":"Total (sum) value for this bucket. Null indicates no data for this bucket.","nullable":true},"label":{"type":"string","description":"Category label for aggregate view data points (e.g. 'Feature', 'Bug'). Only present for githubFeatureVsMaintenance and leadTime when view=aggregate."}},"required":["startDate","endDate"]}}}} ``` ## The ChartResponseDto object ```json {"openapi":"3.0.0","info":{"title":"Multitudes Customer API","version":"1.0.0"},"components":{"schemas":{"ChartResponseDto":{"type":"object","properties":{"metricType":{"type":"string","description":"Metric type identifier."},"from":{"type":"string","description":"Query start date as ISO 8601 timestamp. Absent for aggregate view."},"to":{"type":"string","description":"Query end date as ISO 8601 timestamp. Absent for aggregate view."},"timescale":{"type":"string","description":"Timescale used for data point bucketing. Absent for aggregate view.","enum":["daily","weekly","monthly"]},"options":{"description":"Echoes options that were explicitly provided in the request.","allOf":[{"$ref":"#/components/schemas/ChartOptionsDto"}]},"series":{"description":"Chart series.","type":"array","items":{"$ref":"#/components/schemas/SeriesDto"}},"units":{"description":"Unit of measurement for this metric's values. A plain string for all metrics except aiIntensityOfUsage, which returns an object keyed by value field (spend, inputTokens, linesChanged) since that metric emits three parallel units per data point.","oneOf":[{"type":"string"},{"type":"object","additionalProperties":{"type":"string"}}]}},"required":["metricType","options","series","units"]},"ChartOptionsDto":{"type":"object","properties":{"repositories":{"description":"Repository names the response was filtered to.","type":"array","items":{"type":"string"}},"excludeOrganization":{"type":"boolean","description":"Whether the org-level rollup series was excluded from the response."},"excludeWeekendHours":{"type":"boolean","description":"Whether weekend hours were excluded."},"includeSelfiePrs":{"type":"boolean","description":"Whether self-review PRs were included."},"defaultBranchOnly":{"type":"boolean","description":"Whether only default-branch commits/PRs were included."},"view":{"type":"string","description":"View type for githubFeatureVsMaintenance.","enum":["timeSeries","aggregate"]},"direction":{"type":"string","description":"Feedback direction. Applies to feedbackQuality and feedbackThemes metrics.","enum":["given","received"]},"groupBy":{"type":"string","description":"How to group chart series. Defaults to team. AI metrics support: jobLevel, timezone, aiTool, nonContributors. PagerDuty metrics support: service, escalationPolicy.","enum":["team","service","escalationPolicy","jobLevel","timezone","aiTool","nonContributors"]},"cohortMetric":{"type":"string","description":"Cohort metric used to classify users (aiCohortDistribution only).","enum":["dailyActiveUsage","spend","inputTokens","linesChanged"]},"cohortThreshold":{"type":"number","description":"Threshold value used to split High/Low AI cohorts (aiCohortDistribution only). Unit depends on cohortMetric: percentage of active days for 'dailyActiveUsage', USD cents for 'spend', token count for 'inputTokens', line count for 'linesChanged'."},"cohortLookbackWeeks":{"type":"number","description":"Number of lookback weeks used for cohort classification (aiCohortDistribution only)."},"individual":{"type":"boolean","description":"When true, return one series per team member instead of per team. Supported for meeting-hours calendar, PR cycle, and GitHub code review metrics only.","default":false},"countBy":{"type":"string","description":"Unit to count by. For issue tracker metrics (jiraTypeOfWork, jiraFeatureVsMaintenance, linearTypeOfWork, linearFeatureVsMaintenance): issues (default) or storyPoints. For GitHub feedback metrics (feedbackGiven, feedbackReceived, feedbackParticipationGap): comments (default) or reviews. For page disruption metrics (pagerdutyPageDisruptions, pagerdutyPageDisruptionsOoh): pages (default) or disruptedHours.","enum":["issues","storyPoints","comments","reviews","pages","disruptedHours"]},"priorities":{"description":"PagerDuty priority IDs the response was filtered to (PagerDuty metrics only).","type":"array","items":{"type":"string"}},"urgencies":{"type":"array","description":"Urgency levels the response was filtered to (PagerDuty metrics only).","items":{"type":"string","enum":["high","low"]}}}},"SeriesDto":{"type":"object","properties":{"id":{"type":"string","description":"Series identifier. For team series: team UUID. For cohort series: cohort name."},"name":{"type":"string","description":"Display name for the series."},"type":{"type":"string","description":"Series type.","enum":["team","organization","member","cohort"]},"dataPoints":{"description":"Ordered list of data points for this series.","type":"array","items":{"$ref":"#/components/schemas/DataPointDto"}}},"required":["id","name","type","dataPoints"]},"DataPointDto":{"type":"object","properties":{"startDate":{"type":"string","description":"Bucket start date (ISO 8601)."},"endDate":{"type":"string","description":"Bucket end date (ISO 8601)."},"value":{"type":"number","description":"Metric value. Omitted for comment metrics (commentsGiven, commentsReceived, commentsParticipationRatio) and when percentile data (p50/p75/p90/p95) is present."},"p50":{"type":"number","description":"50th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p75":{"type":"number","description":"75th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p90":{"type":"number","description":"90th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"p95":{"type":"number","description":"95th percentile value. Present when percentile data is available. Null indicates no data for this bucket.","nullable":true},"count":{"type":"number","description":"Number of data points considered in this bucket (e.g. PRs, commits, issues). Null indicates no data for this bucket.","nullable":true},"memberCount":{"type":"number","description":"Number of people considered in this bucket (team members in scope for the data point). Null indicates no data for this bucket.","nullable":true},"spend":{"type":"number","description":"Total AI spend in USD cents for this bucket. Present for aiIntensityOfUsage.","nullable":true},"inputTokens":{"type":"number","description":"Total input tokens consumed for this bucket. Present for aiIntensityOfUsage.","nullable":true},"linesChanged":{"type":"number","description":"Total lines of code accepted for this bucket. Present for aiIntensityOfUsage.","nullable":true},"total":{"type":"number","description":"Total (sum) value for this bucket. Null indicates no data for this bucket.","nullable":true},"label":{"type":"string","description":"Category label for aggregate view data points (e.g. 'Feature', 'Bug'). Only present for githubFeatureVsMaintenance and leadTime when view=aggregate."}},"required":["startDate","endDate"]}}}} ``` # Annotations ## What are annotations? Annotations allow you to add context to your data. Users can annotate chart data points, view annotations in a sidebar, and reply to annotations to have threaded discussions with team members. On line charts, they appear as bubbles on specific data points with a number indicating the number of comments associated with that data point. ## Create an annotation Add annotations in the following ways: * If a chart that doesn’t have any annotations in the current view: click the `Add annotation` link in the bottom-left corner of the chart. * If a chart has annotations visible: click the `View [N] annotations` link in the bottom-left corner of the chart, then click the `+ New annotation` button at the top of the sidebar.
Gif of creating a new annotation

Create a new annotation

Annotations are associated with a specific **chart, date, and team**. They will appear on the relevant data point on the chart, i.e. on the specific series (the team's line, for a line chart), and in the month/week/day data point that the date is included in. For charts that are not grouped by team (anything that’s not a simple line chart with one line representing a team), you will be required to enter a **date range** that the annotation is relevant for. These annotations will not appear on the chart, but just be visible in the sidebar via the `View [N] annotations` link. {% hint style="warning" %} You can only edit and delete your own annotations**.** {% endhint %} ## View annotations in the sidebar
Gif of clicking a bubble on a line chart, which opens the annotations sidebar to show the relevant comment.

Open the annotations sidebar via an annotation bubble

Open the Annotations sidebar by clicking `Add annotation` or `View [N] annotation` on any chart. * The annotations listed in the sidebar are specific to the selected page, and the filters that are set on the page: the date range, and the filtered teams. * Users can `Reply` to annotations, creating threaded conversations. * Click the `Apply filters` link next to `Reply` to reset the filters at the top of the page to what the original annotation author had been seeing\*. * Click the `…` menu next to a comment and select `Copy link` to share a particular comment and the specific filters that were in place when the annotation was made\*. * Click an annotation in the sidebar to highlight the relevant chart. * Click an annotation bubble on a chart to highlight the corresponding annotation in the sidebar.
Gif of clicking overflow menu on a comment to copy a link

Copy a link to a specific comment.

\*This is for you to see the same data that the author was seeing, so that any trends or data numbers that they refer to make sense. ## Notifications [Learn about annotation notifications on our Types of Notifications page.](/knowledge-base/types-of-notifications) # Exporting your data Export your data via API or into a CSV or S3 bucket. Our Multitudes Data Export makes it easy for you to pull a report of your engineering metrics and how they have changed over time. Coupled with our [Engineering Metrics Reporting Template](https://docs.google.com/spreadsheets/d/1X6RBX4xw3sLUERzJRr57iqZvn9YcTLSYm4yanAeJK7U/template/preview), you can turn your metrics into a format that’s easy to share and include in your regular reporting documents. ## **Options for exporting your data** We offer 3 ways to export your data from Multitudes: 1. **API:** this is the most powerful way to export your data. All metrics are included as well as all the drill-down charts. To use this export option, read more here: [Multitudes API](/knowledge-base/multitudes-api) 2. **Export to CSV:** This includes a subset of metrics and outputs a csv. Read below for more: [Export to CSV](#export-to-csv) 3. **Export to S3 bucket:** Like the CSV export, this includes a subset of metrics. It outputs to an S3 bucket of your choice. More here: [Export to S3 bucket](#export-to-s3-bucket) ## **How to export your data** ### Export to CSV 1. On the Multitudes App, go to Settings > [Export Data](https://app.multitudes.co/team-settings/data-export). 2. Select the filters for the data you want to download
* Date range\ **Note:** We recommend selecting full periods, for example 1 Jan - 30 Sept instead of partial periods like 1 Jan - 15 Oct. This will help to ensure the reporting is complete and the trends are more accurate. * Timescale (either ‘weekly’ or ‘monthly’) * Teams * Repositories 3. Request your export 4. Once your data export is ready we will email you - depending on the amount of data this may take a little while to arrive. Data exports should be available within 1 business day, if not please contact us. 5. Follow the link in the email to download your CSV file. \ **Note:** Only the person who requested the export can download the CSV file from the app. 6. Your export will be stored for 30 days in case you need to revisit it and download it later. You can access your previous 30 days of exports in the Requests log at the bottom of the [Export Data page](https://app.multitudes.co/team-settings/data-export). Note: Anyone with Multitudes access can export data from the app.
### Export to S3 bucket {% hint style="warning" %} Note: This functionality is currently in beta, to schedule your report contact . {% endhint %} With our Export to S3 functionality, you can set up a report to write directly to your specified S3 bucket on the third day of every month, at 1200 UTC. You can see details for this on the [Export Data](https://app.multitudes.co/team-settings/data-export) page. Note: We only support exports to one S3 bucket. So any future exports created to write to S3 will need to use the same configuration details. #### Setting up an export configuration You will need to provide the following details: * Bucket Name * AWS Region * AWS Access Key ID * AWS Secret Access Key #### Scheduling a report The report will display the previous full month's data. You can schedule multiple reports to the S3 bucket, the information we require is: * Timescale (daily / weekly / monthly) * Teams * Repositories {% hint style="info" %} **Note**: S3 exports automatically [exclude weekend hours](/metrics-and-definitions/process-metrics/flow-of-work#filtering-flow-of-work-metrics) from time-based flow metrics (Change Lead Time, Review Wait Time, etc.). If you need data that includes weekend hours, please use the CSV export option instead. {% endhint %} ## **What’s included in your export** All Multitudes time series metrics are included in your export. For each metric, we export the data available in the app so you have the flexibility to analyze it as you need. For example: * **For Change Lead Time metrics** (including Coding Time, Review Wait Time, Editing Time and Deploy Time) we include P50 (median), P75, P90, and P95 percentiles. * **For Merge Frequency**, we export all PRs per person; all PRs total; collaborative PRs per person; and collaborative PRs total.
**MTTR and MTTA:** In the Multitudes app, these numbers are grouped by service and escalation policy. In the export, we show one aggregate value over time per team. **Weekend hours handling:** * **CSV Export**: Includes both versions of time-based metrics - [with weekend hours included AND with weekend hours excluded](/metrics-and-definitions/process-metrics/flow-of-work#filtering-flow-of-work-metrics). You can choose which version to use when analyzing the data. * **S3 Export**: Excludes weekend hours by default for all time-based flow metrics. **Timezone:** Multitudes data is displayed in your organization's time zone. **Deployment data source:** If an organization has both the Deployments API AND GitHub Actions set up, this tool will prioritize the Deployments API data and show that over the GitHub Actions data. **Team data only:** This export is designed to be used to report on team data, not individual data. ### Multitudes metrics not included: Some Multitudes charts aren't included because they're not time series data. The following metrics are not exported: * Value Delivery: Feature versus Maintenance chart (from Linear, JIRA, or conventional commits data) * Quality of Work: Number of pages * Collaboration: Feedback Flows **Take action tools not included:** The Multitudes app also has other functionality not shown in the data export – the insights, actions, facilitation guides, and custom targets are all excluded. ## **Using your exported data with our reporting template** It’s likely that you’ll have a lot of data in your export! The way that you share and analyze it is important, so we’ve created a tool to help you turn the raw CSV export of your engineering metrics into a format that's easier to read and share. We've made a public Google Sheet template that takes in your exported CSV data from Multitudes and presents an output table showing how metrics have changed over time for different teams. {% hint style="info" %} Note: The template includes a toggle option to choose between [including or excluding weekend hours](/metrics-and-definitions/process-metrics/flow-of-work#filtering-flow-of-work-metrics) for flow metrics, matching the options available in your CSV export. {% endhint %} [**View the template**](https://docs.google.com/spreadsheets/d/1X6RBX4xw3sLUERzJRr57iqZvn9YcTLSYm4yanAeJK7U/template/preview) 1. **Make a copy of the Google Sheet template above.** 2. **Paste your exported data from Multitudes into the "INPUT\_csv data” sheet. The CSV download from our app will be in the correct format already.**\ If you are using a different CSV, check the template readme for details on setting up your data. 3. **Review your metrics in the "OUTPUT" sheet**\ You data will be presented in a table in the "OUTPUT" sheet. Use the drop down selectors at the top of the sheet to choose what you need. You can filter between teams, normalize data to per-person, and set toggles for Flow of Work and Collaboration options. The "OUTPUT" sheet includes: * A link to read about each metric and why it’s measured in our Help Docs * The unit of measurement * A benchmark * A sparkline reflecting the change over time * Data is colour coded to show changes # Types of Notifications ## Get insights and notifications in Slack Multitudes can proactively send you notifications to help you improve performance, collaboration, and wellbeing, in your existing workflow. ## Explore our Notifications
PR Updates – ad hoc/pages/odB5zJD1A6dozBEdVjGz
Work Digest – daily/pages/izWYNFa7lVE7O6aekHRf
Trend Summary – weekly/pages/olXjslxCTaD0S8iYg2e4
1:1 Prompts/pages/DiwQ5SsvM51AP5rJXUeP
Annotations/pages/TwThXenFIHXqP69ObqPl
If you haven't set up our Slack integration yet, please head to our [Slack integration page](/integrations/slack) for more information. You can find more instructions on how to configure your email and slack notifications in our [Notifications Configuration page](/configuration-and-setup/notifications-configuration). # PR Updates Ad hoc notifications when a review is requested from a team and/or individual. Stay on top of pull request activity without leaving Slack. Multitudes notifies your team at every stage of the PR lifecycle – from review requests and comments to approvals and merges.
## How it works When a review is requested on a PR, Multitudes sends a notification to the relevant Slack channel and/or DM (depending on your configuration). As the PR progresses, the topline message updates, with each action on the PR being shown in the message thread. Here's how PR updates compare to GitHub's Slack integration: * **Less noisy thanks to thoughtful threading**: The topline message in the Slack thread updates to always show the latest PR status, with previous events showing in the thread. This keeps the Slack channel clean, and only notifies you for events that need your attention (like review requests). * **Shows PR information, so you can estimate effort:** PR updates includes more information about the PR (like the size and PR description), so you can get a quick idea of how much effort it might take you to review. * **More configurable:** Every person and team has different preferences about what they want to be notified by. PR updates can be configured at the team and individual level, with per-event toggles for each person (review requested, approved, changes requested, comments, mentions, merges), so people only get notified about what's actually relevant to them. ## Who it's for Turn on PR updates if you want to keep PRs flowing smoothly – this can help reduce waiting time and, overall, your [Change Lead Time](/metrics-and-definitions/process-metrics/flow-of-work/change-lead-time). These notifications give teams a place where they can easily see the status of all the PRs their team is working on, and the same for individuals who want to know where their individual work is at. PR updates work at two levels, and most teams use both together: * [Team PR updates](#team-pr-updates): Ad hoc notifications when a review is requested from the team. * [Personal PR updates](#personal-pr-updates): Ad hoc notifications for changes to a PR that an individual has been involved with (based on their configuration for what they want to be notified about). Read on below for more about each. ### Team PR updates These ad hoc notifications are based on PR activity relevant to the team – for example, when the team or a team member is asked to review a PR. How it works: * Each review request starts a new thread * Configure if the team should be notified about review requests to the team and/or to any individual on the team * The review request notification includes information about the PR, including size and the PR description, so you can determine how much effort it might take to review * As the PR changes, updates go into the thread and the topline message changes to show the current status of the PR * Once a PR no longer needs action (it's merged, closed, or moved to draft state), the toplevel message is crossed out. To set these up, go to [Team PR updates setup](#for-team-pr-updates) below.
### Personal PR updates These ad hoc notifications turn PR activity into Slack DMs to an individual. You can configure this for the types of activity you care about, such as: * Activity on a PR you've authored – including approvals, changes, comments, or merges (from someone else) * Activity on PRs you're involved with – including review requests, @-mentions, or merges Similar to the Team PR updates (see above), each new PR with activity starts a new topline message, additional activity goes into the topline message and gets posted into the thread, and PRs that don't need any further activity are crossed out. To set these up, go to [Personal PR updates setup](#for-personal-pr-updates) below.
## How to set up PR updates ### For Team PR updates Set up Team notifications to get notified about review requests to a team. These notifications will be sent to the Slack channel that you specify. 1. Go to [**Settings > Notifications > Team**](https://app.multitudes.co/team-settings/notifications?section=team). 1. If you see a button with "Connect your Slack organization", then you'll need to get a Slack admin to install Slack with the instructions here: [**Integrations > Slack**](/integrations/slack) 2. If you see a table with team names (like the below), then that means Slack is installed for your organization and you can proceed.
2. You will see all your teams listed with a column on the right for specifying the slack channel where you want to receive PR Update notifications. 1. Click on the Edit icon on the right for the team that you want to receive notifications for. 2. Under **Team PR updates**, toggle **Send via Slack** to **ON.** 3. Search for and select the slack channel where you want to receive the notifications. 4. Tick whether you want to include PRs where the team is a code-owner. 5. Tick whether you want to include review requests from bots and agents. 6. Click **Save**. **Note: To get notifications in private channels**, you need to add the `Multitudes for Slack` app to those specific channels. You can do this by typing `@multitudes` in each private channel, or clicking on the private channel's name and going to `Integrations > Add apps`. ### For Personal PR updates You can set up personal notifications to get notified about PRs that you author or are involved with. 1. Go to [**Settings > Notifications > Personal**](https://app.multitudes.co/team-settings/notifications?section=personal). 1. If you see a button with "Connect your Slack DMs", then you'll need to install Slack for your individual account following the instructions here: [**Integrations > Slack**](/integrations/slack) 2. If you see a page with Slack notification options (like the below), then that means Slack is installed for you and you can proceed.
2. Go to the **PR updates** section (at the top of the page). 1. Toggle **Send via Slack** to **ON**. 2. Tick the activities that you want to get notified about for PRs you authored as well as PRs you're involved with. 3. Click **Save.** # Work Digest Daily notifications providing a summary of work in progress. Get a daily summary of your team's work and what needs attention – including how much work was completed since the last update, what the AI usage has been, how much work is being done out-of-hours, and what pull requests are awaiting action (be it a review, changes to the code, or a merge). {% hint style="success" %} We recommend scheduling these to arrive at the start of stand-up, so you can discuss as needed there. {% endhint %} Here's an example:
### 1st section: Since the last summary... * **Work completed:** The number of PRs merged and the number of tickets moved to done since the last digest. * **AI:** Your team's [Daily Active Usage](https://docs.multitudes.com/metrics-and-definitions/ai-impact-feature/ai-adoption#daily-active-users-daus) and [Intensity of Usage](https://docs.multitudes.com/metrics-and-definitions/ai-impact-feature/ai-adoption#intensity-of-usage) of AI tools. * **Wellbeing:** You team's wellbeing shown as number out-of-hours commits. These values are filtered to just the people on the selected team, since the last digest was sent. ### 2nd section: PRs awaiting review Header: **"\[X] PRs are awaiting review."** These are open PRs that don't yet have any reviews – a good opportunity to nudge someone to review this work. Each PR shown includes: title, PR author, how long it's been waiting for a review – with a ⏰ if it's been waiting for more than 8 hours – and what the last action was, if there's been an action since moving to review requested (e.g., who last commented). Note that Slack automatically truncates the list if it gets long; you can click "+N more" to see the rest. ### 3rd section: PRs with changes requested Header: **"\[X] PR(s) had changes requested."** These have had a review, and are now waiting on the author to address the feedback. The PR information is the same as in the "PRs awaiting review" section, with the note that the time waiting is the time since the PR had changes requested. ### 4th section: PRs approved Header: **"\[X] PR(s) has been approved."** These are ready to merge. The PR information is the same as in the "PRs awaiting review" section, with the note that the time waiting is the time since the PR was approved. {% hint style="info" %} The list excludes: * PRs that haven’t been updated in more than 7 days, to avoid surfacing stale PRs day after day * Draft PRs * PRs with the string \[do not merge] in the PR title (case insensitive) {% endhint %} ## How to set it up in Slack 1. Go to [**Settings > Notifications > Team**](https://app.multitudes.co/team-settings/notifications?section=team). 1. If you see a button with "Connect your Slack organization", then you'll need to get a Slack admin to install Slack with the instructions here: [**Integrations > Slack**](https://docs.multitudes.com/integrations/slack) 2. If you see a table with team names (like the below), then that means Slack is installed for your organization and you can proceed.
2. You will now see all your teams. 1. Click on the Edit icon on the right for the team that you want to receive notifications for. 2. Under **Work digest**, toggle **Send via Slack** to **ON.** 3. Search for and select the Slack channel where you want to receive the notifications. 4. Choose when you want to receive them and if you want to include reviews from bots and agents. 5. Click **Save**.
# Trend Summary Weekly notifications with insights about key metrics and recommended actions. This notifications provides an overview of how the selected team/organization is doing. We show 5 key metrics in this: * `Change Lead Time` * `Merge Frequency` * `Change Failure Rate` * `Out-of-Hours Work` * `Participation Gap` ## Description of each section

Trend Summary notification example

* **Check in on:** Metrics that have recently gotten worse or have been consistently doing poorly * **Improved:** Metrics that recently improved * **Still on track:** Metrics that have been consistently trending well Each section will appear only if there is at least one metric in that section. The section will be removed if there are no metrics that apply, which means that your Trend Summary Notification may appear different from what is shown on this page. For each metric, we give you the insights, and then an action if the insight status is potentially concerning. The status of these [insights](/metrics-and-definitions/multitudes-insights) are based on targets that you can [customize](/configuration-and-setup/customize-targets) from our default industry benchmarks. {% hint style="info" %} For more information on these 5 metrics including how they’re calculated and the research behind our default industry benchmarks, check out our page on [Metrics & Definitions](/metrics-and-definitions/multitudes-insights). {% endhint %} ## Who it's for Turn on Trend Summary notifications if you want regular updates on how your team is trending, delivered in your workflow. It's a quick view across 5 key metrics, and you can configure how you receive these notifications. There are two ways you can receive these notifications: * **Team Trend Summary:** Weekly notifications with insights and recommended actions, delivered to the whole team via Slack. You specify which team should get updates and in which Slack channel. Setup instructions here: [Set up Team Trend Summary](#for-team-trend-summary-notifications) * **Personal Trend Summary:** Weekly notifications with insights about key metrics and recommended actions, for teams you are part of or [watching](https://docs.multitudes.com/configuration-and-setup/permissions#watcher-role). These notifications are sent to your email. Setup instructions here: [Set up Personal Trend Summary](#for-personal-trend-summary-notifications) ## How to set up Trend Summary notifications ### For Team Trend Summary notifications 1. Go to [**Settings > Notifications > Team**](https://app.multitudes.co/team-settings/notifications?section=team). 1. If you see a button with "Connect your Slack organization", then you'll need to get a Slack admin to install Slack with the instructions here: [**Integrations > Slack**](/integrations/slack) 2. If you see a table with team names (like the below), then that means Slack is installed for your organization and you can proceed.
2. You will see all your teams listed with a column on the right for specifying the Slack channel where you want to receive Trend Summary notifications. 1. Click on the Edit icon on the right for the team that you want to receive notifications for. 2. Under **Trend summary**, toggle **Send via Slack** to **ON.** 3. Search for and select the Slack channel where you want to receive the notifications. 4. Select when and how often you want to receive the notifications. 5. Toggle whether you want to exclude weekends. 6. Toggle whether you want to only count PR reviews. 7. Click **Save**. **Note: To get notifications in private channels**, you need to add the `Multitudes for Slack` app to those specific channels. You can do this by typing `@multitudes` in each private channel, or clicking on the private channel's name and going to `Integrations > Add apps`. ### For Personal Trend Summary notifications 1. Go to [**Settings > Notifications > Personal**](https://app.multitudes.co/team-settings/notifications?section=personal). 2. Under **Trend Summary**, toggle **Send via Email** to **ON**
# Multitudes AI Coach Our AI Coach shows the top insight for your team(s) at the start of each week, and also lets you bring up insights on demand via slash commands. It analyses and understands at each team’s data and context , and then helps flag anything that looks concerning, suggest contributing factors, and give ideas for actions to take (e.g., questions to ask the team for more context). The results also include deep links into specific charts and views around the app, for you to dive deeper into the details.
screenshot
## Slack slash commands Alongside the weekly insights, you can also use slash commands to get insights on your team from our Multitudes AI Coach at any time. {% hint style="info" %} Note that even if you type these commands outside of a direct message with the Multitudes App (e.g. in a public channel), Multitudes App will send the results to you in a direct message. You can forward this to a public channel. {% endhint %} Type `/multitudes help` for the menu of options, or check out the list of parameters below: * ‍`/multitudes` with no parameter or `/multitudes insights`\ Brings up the AI Coach menu, which walks you through a flow to learn more about the metrics listed below * `/multitudes blocked`\ Which PRs are blocked for people in your team(s) * `/multitudes lt` or `change lead time` or `flow`\ [Change Lead Time](/metrics-and-definitions/process-metrics/flow-of-work/change-lead-time), how long it takes for PRs to go from first commit to deploy, our key metric for flow of work * `/multitudes mf` or `merge frequency` or `delivery`\ [Merge Frequency](/metrics-and-definitions/process-metrics/value-delivery/merge-frequency), the number of PRs merged over the past week, our key metric for value delivery * `/multitudes cfr` or `change failure rate` or `quality`\ [Change Failure Rate](/metrics-and-definitions/process-metrics/quality-of-work/change-failure-rate), the % of PRs that indicated a fix to a previous failure, our key metric for quality of work * `/multitudes pgap` or `participation gap` or `collaboration`\ [Participation Gap](/metrics-and-definitions/people-metrics/collaboration/pr-participation-gap), the gap between the largest and smallest share of voice on your team(s) based on PR comments, our key metric for collaboration * `/multitudes ooh` or `out of hours` or `wellbeing`\ [Out-of-hours work](/metrics-and-definitions/people-metrics/wellbeing/out-of-hours-work), commits made outside of an individual's preferred working hours, our key metric for wellbeing # 1:1 Prompts This shows an idea for a conversation starter at your next 1:1 with a specified team member, based on the team member's data. ## **Who is included in a 1:1 prompt?** Anyone who you’ve added to the 'Who you have 1:1s with' section of your profile under Settings. ## **How can I share this with my team member?** Make sure your team member has also connected their Slack account to the Multitudes app via the Settings > Integrations page. This means that the team member must have [login access](/configuration-and-setup/permissions-and-roles) to the Multitudes app. Once they’ve done that, they will automatically start receiving a copy of the conversation starters.
Screenshot

1:1 Prompt notification example

# Annotations notification
Screenshot of Slack notification about someone making an annotation on a chart.
You will receive notifications relating to [our Annotations feature](/knowledge-base/annotations) via our [Slack integration](/integrations/slack) if you have it installed, or to your email. Notifications will be sent when: * Someone replies to an annotation you've made * Someone adds an annotation on your individual data (e.g. your series on the "Individual" view for a chart like `Review Wait Time`) {% hint style="warning" %} There currently is no way to configure these notifications - this is in progress. {% endhint %} # Troubleshooting Missing Commits ## My commits don’t seem to be appearing in Multitudes – what’s going on? Check that the email address that is in your local git config (`user.email` when you do `git config -l`) matches the email address(es) that are linked to your GitHub account. You can change this by either changing git config to match an email that’s in GitHub, or by adding your git config email address to your GitHub account under . {% hint style="warning" %} If the emails are different, GitHub won’t know how to match your commits to your GitHub login (although it still links it to the account because of your SSH keys). {% endhint %} # Bot Activity We currently filter out bot activity because it's useful to see patterns of how humans interact with other humans. However, we have work coming up soon to give you more control over (1) What counts as a bot (bot = any actor that's not a human, so this includes your AI bots/agents) (2) When you want to include/exclude bots from your metrics If you're interesting in giving feedback on this, reach out at . In the meantime, read on below for more about how bots are handled in the app today. ### What we consider a bot We consider an account to be a bot if it meets any of the following: * A user that GitHub deems is of type `Bot` or `Organization` * A user of type `User` with the string `-bot`, `[bot]`, or `dependabot` in the name (case sensitive). E.g., `bertie-bot` and `bertie-bott` would be considered bots, but `hannah-abbott` wouldn't. * A user of type `User` with a name that exactly matches one in our specified list. This list includes common GitHub bots who come through with a type of `User`, like `netlify`, `linear-app`, `renovate`, `renovate-approve`, `github-actions` – as well as common AI bots/agents like Code Rabbit, Copilot, and more. ### What we exclude Once an account is identified as a bot, we exclude the following: * PRs authored by bots. * A PR with `[snyk]` in the title (case insensitive). * Reviews and comments left by bots. This means that these reviews and comments: * Won't count towards PR review counts * Won't be included in [Review Wait Time](https://docs.multitudes.com/metrics-and-definitions/process-metrics/flow-of-work/review-wait-time) or other review metrics * Won't show up in the [Feedback Quality](https://docs.multitudes.com/metrics-and-definitions/people-metrics/collaboration/feedback-quality)/[Feedback Themes](https://docs.multitudes.com/metrics-and-definitions/people-metrics/collaboration/feedback-themes#how-we-calculate-themes) charts {% hint style="info" %} If you notice a bot account showing up in your metrics that you believe should be filtered out, let us know at . {% endhint %} # Collaborative PRs & All PRs Toggles On the Merge Frequency chart on the [Value Delivery](https://app.multitudes.co/value-delivery) page, you’ll see an option to choose between `Collaborative PRs` or `All PRs` in the top-right corner of the chart. This toggle lets you only see PRs that had input from other team members (not just the PR author), like a comment, a review, or a merge. * When `Collaborative PRs` is selected (default), only PRs that have had a comment, review, or merge from someone other than the PR author are shown; in other words, “selfie PRs" are excluded. * When `All PRs` is selected, all PRs are shown - even the “selfie PRs" where there was no comment, review, or merge on the PR by anyone other than the PR author. # Why we use percentiles What percentiles are and why we use them We talk a lot about percentiles in Multitudes, so this page provides an intro to percentiles – what they are, why they're useful, and which ones you should use for what. To set your default percentiles in Multitudes, jump to this section: [How to set preferred percentiles](#how-can-i-set-my-preferred-percentile). ## What are percentiles? A percentile tells you what percentage of the data sits under a given number. If the 75th percentile (P75) for age in a room of 100 people was 39, that means 75 people in the room are 39 or younger. Some percentiles have special names: * P25 = first (or lower) quartile * P50 = median, or second quartile * P75 = third (or upper) quartile Percentiles give you a clearer picture of your data's shape, because you always know how many observations sit above or below a given point.
## Where do you use percentiles in Multitudes? Charts for certain metrics in the app show the data aggregated to percentiles instead of the average: * `Change Lead Time` * `Coding Time` * `Review Wait Time` * `Editing Time` * `Deploy Time` * `Lines of Code Changed (PR Size)` * `Files Changed (PR Size)` You can set your preferred percentile for these charts to be P50, P75, P90, or P95 (we recommend P75, or P50 if you're just getting started). See more here: [How to set your preferred percentile](#how-can-i-set-my-preferred-percentile). ## Why not just use the average (mean)? Both the mean and the median describe what's "typical" in a dataset (they're both [measures of central tendency](https://www.abs.gov.au/statistics/understanding-statistics/statistical-terms-and-concepts/measures-central-tendency)) – but they respond very differently to outliers. The **mean** (or "average") is calculated by averaging all the numbers together: sum all the numbers and divide by the count of numbers. Because the value of every number is included in the sum, the mean is susceptible to outliers. For example, if your largest number is 10x the next-highest value, that will drag the mean up. The **median** doesn’t have that issue. It’s rank-ordering the numbers, so whether your largest number is 10x the next-highest or just 1.1x the next-highest, the rank doesn't change – and so the median likely won't be impacted. As long as you have 3+ numbers in your dataset\*, one outlier won't change the median. Another benefit of using the median is that you always know that 50% of the values sit above it and 50% of the values sit below it. This makes it much better for making real-world decisions. For example, say you want your team's code changes to get a human review in under 4 hours. If you measured that goal with the average, one review that takes weeks would drag the average up, even if most reviews take under 4 hours. That single outlier would make the goal look further out of reach than it really is. Your P50 or P75, on the other hand, won't move if your dataset is big enough (in this case, at least 6 numbers.) *\*Note:* If you only have one or two numbers in your sample and one is very large, that will impact the median. When you have one value, the median is that value – and for a dataset of 2, you'll average the two values to get the median. That said, it's hard to draw any conclusions from datasets that are this small anyway. ## How do you calculate a percentile? First, sort the values in your dataset from smallest to largest. There are several accepted formulas for calculating percentiles, but the core idea is the same: cut the sorted data at the relevant point. For quartiles, you split the data into 4 equal groups; for deciles, into 10 equal groups. A percentile is just a more general version of this – it tells you the cut point for any percentage you choose. For more about how to calculate percentiles, read our blog post here: [What are percentiles – and why we recommend P75](https://www.multitudes.com/blog/what-are-percentiles-and-why-we-recommend-p75) ## What percentile should you use? This depends on your goals. For Multitudes metrics, we generally recommend P75. Here's why: * P75 gives you confidence that most datapoints are under that level – because 75% of the numbers will be less than that. If P75 looks good, most of your work looks good. * That said, the median (P50) can be a good starting point for teams early in their goal-setting journeys who want to start with something more achievable – since P50 just looks at what half the data did. P75 walks a nice line between showing you what most (75%) of the work looked like, while still leaving room for outliers. ## How can I set my preferred percentile? Your options depend on your permission level in Multitudes: * [Owners and Managers](#owners-and-managers) can set the organization default * [Anyone](#everyone) can update the percentile they see when they log in ### Owners and Managers Owners and Managers can set the organization default. * Setting a default percentile for your organization will determine with percentile is used in Multitudes's notifications, and will set the default when a team member first logs into Multitudes. * To set this, go to [Settings > Organization Settings > Default settings](https://app.multitudes.co/team-settings/organization-settings?section=default-settings) and choose your preferred percentile. (Screenshot below.)
### Everyone Anyone – from Members to Managers and Owners – can update the view they see when they log into the app. We use saved filters in Multitudes, which means that we'll save your most recent filters and load those the next time you visit the app. So to change your percentile: 1. Go to one of the pages that has a filter for percentiles: The [Flow of Work](https://app.multitudes.co/flow-of-work) or [AI Impact](https://app.multitudes.co/ai-impact/impact) pages 2. Find the "Percentile" filter at the top of the page and choose your preferred percentile (screenshot below). We'll then show you this percentile the next time you log in.
# Billing & Payments Learn about our pricing and how to manage your billing and payments. ## **What pricing plans do you offer?** We offer both Standard and Enterprise plans with different features. For detailed pricing information and plan comparisons, please visit our [Pricing Page](https://www.multitudes.com/pricing). ## **Who do I get charged for on the Multitudes app?** Payment is charged based on [data inclusion](/configuration-and-setup/permissions-and-roles). In short, you are only charged for Contributors, people whose data is included for analysis (e.g., PRs, comments, etc.), and you are not charged for Viewers whose data isn’t included but who can login to the app. One thing to keep in mind is that if you [sync your teams to GitHub](/configuration-and-setup/adding-users-and-teams#automatically-adding-team-members-via-github-teams-sync), we will automatically add and remove Contributors, and notify you afterward. ## Can I try it for free? Yes! We offer a 2 week free trial. When your trial ends, you'll need to provide payment information to continue using Multitudes. We'll send you a reminder before your trial expires. More info on our [Pricing page](https://www.multitudes.com/pricing). ## When do I need to pay? You can start a free trial without providing any credit card information. If you want to continue after your trial, we’ll ask for your payment information then. ## **Which payment methods do you accept?** Payments (whether monthly or annual) can be made via credit card. Enterprise annual plan customers with a minimum yearly invoice of $20,000 USD have an additional option to pay via bank transfer. Note that only account "Owners" can access the Billing page in the Multitudes app to manage payment information. ## Do you store any credit card information in your systems? No. We use Stripe to manage credit card payments. You can learn more about their security practices [here](https://stripe.com/docs/security/stripe). ## **How does billing work if I add or remove users?** You can add or remove Contributors at any time during your billing cycle. For monthly plans, any changes will be automatically prorated and reflected in your next monthly invoice. For annual plans, whenever you add a new contributor, you’ll be invoiced upfront for the cost of that contributor to the end of your current 12-month payment cycle. If you remove contributors on an annual plan, you’ll get prorated credits which Stripe will automatically apply to your next bill. ## Can I change or cancel my plan at any time? You can cancel your subscription whenever you like; however, we do not offer refunds for a billing period that you’ve already paid for. ## **Who can I contact for billing support?** For any questions, please contact our support team at # Security & Privacy Get all the information you need about our key security practices. ## Do you ingest our source code? What data does Multitudes get? No, our app does not ingest your codebase – instead, we look at code metadata from GitHub. This metadata includes information about pull requests (such as when they were created, who the author was, commits made on the pull request, number of lines changed, etc.) and about comments (including comments written on pull requests and reviews submitted). We also pull in the contents of comments and reviews. When you set up the GitHub installation, you can see the full list of permissions that Multitudes requests. For Github Actions, we ask for read-only access to Deployments and Environments to get events on a deployment and so we can determine which deployment environments you have set up for each repository. This gives us access to list Deployment Environments in GitHub's REST API as described [here](https://docs.github.com/en/rest/deployments/environments?apiVersion=2022-11-28#list-environments) as well as the ability to subscribe to deployment events as described [here](https://docs.github.com/en/webhooks-and-events/webhooks/webhook-events-and-payloads#deployment). Note that this **does not grant us access to secrets or variables** contained in your environments, as per GitHub's [documentation](https://docs.github.com/en/rest/actions/secrets?apiVersion=2022-11-28#about-secrets-in-github-actions) on Secrets: GitHub Apps must have the secrets permission to use these endpoints. This is something Multitudes would never require or ask customers for. ## How safe is our data? We keep your data secure by using the latest cloud technologies and security principles. All data is encrypted at rest and in transit, with strict access control as to who can see what data. We only store the minimum data that we need to provide insights for you and your team. For more information, check out our [Security](https://www.multitudes.co/security) page. ## How will this protect the privacy of GitHub users and my team? Most of the data that Multitudes shows is already visible to team members; Multitudes aggregates the information and shows it in new ways.\ \ Multitudes may show individual insights about Collaboration and Wellbeing because those insights are useful for supporting individuals. However, Multitudes limits the detail it shows about performance by aggregating this data so it’s not shown by individual. This is because PRs are a team sport, so it’s important to focus on team performance over individual performance. We do this both to protect the privacy of individuals and to discourage users from making reductive decisions using Multitudes (since Multitudes is only one measure of a team member's contributions to a team).\ \ For more information, please see the latest privacy policy on the Multitudes website [here](https://multitudes.co/privacy-policy). ## Who can see my individual performance data? Our [data ethics principles](https://www.multitudes.co/blog/measure-what-matters-and-dont-be-creepy) guide us to use data to empower teams – to support them to make better decisions for themselves. Code is a team sport, and a [10x team](https://www.multitudes.co/blog/forget-the-10x-developer-focus-on-the-10x-team) is far more important than a lone 10x developer. That’s why we don’t show individual performance metrics – because they’re not the outcome we need to solve for. (If you’re curious, this blog post shares more about [Metrics & Definitions](/metrics-and-definitions/multitudes-insights).) We know that it’s tempting to look for a quick or “simple” answer on how team members are contributing – but unintended consequences are a reality. Individual performance metrics often encourage people to game the numbers or make overall simplistic inferences about performance. Both of those contribute to a broader environment where people optimize for themselves instead of for the team – the opposite of the outcome we want. The result of this can be detrimental to delivering against business goals and customer value. This is also why our app encourages and even suggests questions to explore with the team to get more context on the data. The only time we do show individual metrics is when they meet our data ethics principles – specifically, that the likelihood of using the metric to support people is higher than the likelihood of it being used to harm. Wellbeing and collaboration metrics like Review Wait Time, Out-of-Hours Work, or PR Feedback Given show who’s waiting too long for feedback, who’s at risk of burnout, and who’s doing the glue work to support others on the team (respectively). We do regular reviews of new features against our data ethics principles, and we welcome feedback on our approach – so feel free to [get in touch](https://www.multitudes.co/contact) if you have thoughts! ## Can I (as an individual team member) opt out? Yes, you can. Just email and we’ll take care of that for you. If you opt out, we won’t show your individual data in the Multitudes app and you’ll no longer have access to the app. For teams where one person wants to opt out, we do recommend that the whole team have a conversation about whether they should be using Multitudes. Our goal is to support team collaboration, and so we think it's best that the team make a unified decision about whether or not to use our product. ## What will you do with our data if we cancel our plan? If your organization cancels your plan, we will keep your data for 30 days. After that, we will delete all data for your organization.