AI effectiveness

The AI effectiveness report uses your agent session data to show how effectively your organization is working with AI agents. Use this report to understand where agents are encountering friction, what kinds of work agents are doing, and which sessions explain the patterns you see.

Note: This report requires AI Code Insights with transcripts enabled. Adoption, velocity, and quality metrics also require DX AI, enabled AI tool connectors, and SCM data.

AI effectiveness report in DX

When to use AI effectiveness

AI effectiveness answers the question: How are engineers using AI agents, and where are agents getting stuck?

This report helps teams:

  • Find agent friction - See where sessions are slowed by unclear requirements, unhelpful steering, or output that drifts from the original goal.
  • Understand agent work - See what types of work agents are doing across your organization, broken down by work type and category.
  • Inspect session evidence - Drill into the sessions behind a friction area, work type, team, or group to review the session details, output, and transcript.
  • Compare patterns across teams - Break down the report by team, group, or attribute to see where agent workflows are working well and where teams need support.

What the report shows

The report combines high-level organization patterns with session-level drilldowns.

Agent experience

The Agent experience section shows how effectively engineers are working with AI agents. It includes the Agent Experience Score, a mean score across evaluated AI coding sessions shown out of 5.

Friction areas

Friction areas show where agents encountered the most difficulty. Each bar counts AI sessions rated 3 or lower out of 5 on an Agent Experience dimension. Click a friction area to open the related sessions.

Agent work

Agent work shows how AI sessions break down by the type of work agents are doing. You can view the data by Work types or Categories. A single session can be assigned up to three work types. Click a work type or category to open the related sessions.

Session drilldowns

Drilldowns connect each high-level pattern to the sessions behind it. You can drill into a friction area, work type, category, or breakdown row.

Session details can include:

  • Overview - A summary of what happened during the session and how it was evaluated.
  • Output - The repositories, branches, commits, pull requests, and deployments associated with the session when available.
  • Transcript - The full message history for the session when transcript access is enabled.

Use session drilldowns to separate systemic patterns from one-off sessions. For example, a high count for Poor upfront requirements or context provided can point to issue templates, planning practices, or prompt guidance that need improvement.

Breakdown by team or group

The breakdown table shows how AI effectiveness patterns vary by team, group, or selected attribute. Use the aggregation selector to change the breakdown, and use the date range, team, group, and attribute filters to narrow the report.

When agent session data is available, the table includes Agent Experience Score. When AI Effectiveness Score data is available, the table also includes Overall score, Trend, Adoption, Velocity, and Quality.

The AI usage attribute group is excluded from the attribute filter because the report already uses it to identify AI users for Velocity and Quality.

AI Effectiveness Score

The top of the report can show Overall score, Adoption, Velocity, and Quality. These metrics add context about whether AI usage is paired with broader engineering outcomes.

The Overall score is calculated as:

(Adoption score + Velocity score + Quality score) / 3

Where:

  • Adoption score measures the share of eligible contributors actively using AI tools, based on AI tool provider connector data.
  • Velocity score measures PR TrueThroughput for users with AI tool usage, based on AI tool provider and SCM connector data.
  • Quality score measures the PR defect ratio for users with AI tool usage, based on AI tool provider and SCM connector data.

Each component is normalized to a 0-100 score before the average is calculated. For Quality, lower PR defect ratio produces a higher score. If any component is missing, the overall AI effectiveness score appears as N/A.

The Velocity and Quality components only include users tagged as light, moderate, or heavy AI users in DX’s system AI usage group.

How AI effectiveness differs from AI impact

Unlike AI impact, AI effectiveness focuses on how AI agents are being used in day-to-day engineering work. It shows the friction agents encounter, the work they are doing, and the sessions behind those signals.

Use AI effectiveness to improve agent workflows and coaching, and AI impact to analyze AI-related productivity patterns, time savings, and financial impact.