Agent Experience Score
The Agent Experience Score report evaluates how clearly eligible coding-agent sessions were framed, steered, and scoped. Use it to find recurring interaction friction, inspect representative sessions, and choose a specific workflow improvement.
Note: This report requires AI Code Insights and Transcripts & Agent Experience enabled by a DX admin. Transcript access for each viewer is controlled separately.
When to use Agent Experience Score
Agent Experience Score answers the question: Where do captured coding-agent sessions repeatedly encounter interaction friction?
This report helps teams:
- Find a recurring failure mode — Compare requirements, steering, and scope across teams, contributors, attributes, and tools.
- Inspect the evidence — Open the sessions and evaluation comments behind a score.
- Target one intervention — Improve task context, issue templates, agent guidance, Skills, configuration, or task sizing.
- Remeasure the same cohort — Check whether the pattern changes after a documented intervention.
Do not use the score as a code-quality grade, employee ranking, developer-satisfaction measure, or ROI calculation.
How Agent Experience Score is calculated
After an eligible session ends, DX evaluates its scrubbed transcript with a separate model. The coding agent used during the session does not self-report the score.
Under the current rubric, an evaluated session receives a 1–5 rating for three dimensions. Its session score is:
(Requirements + Steering + Scope) ÷ 3
The report can aggregate session scores using mean, median, p10, p25, p75, or p90. Confirm the selected calculation before comparing a chart or breakdown.
Older stored sessions can include legacy evaluation dimensions. Sessions with insufficient evidence, an unreadable transcript, or unsupported length can remain unevaluated. Missing scores should not be treated as low or high scores.
Scoring dimensions
| Dimension | What the evaluation considers | A low score can point to |
|---|---|---|
| Requirements | Whether the goal was clear and the initial context helped the agent begin | Missing acceptance criteria, unclear goals, or insufficient repository and domain context |
| Steering | Whether follow-up input helped guide progress | Ambiguous corrections, delayed redirection, or feedback that did not resolve the blocker |
| Scope | Whether the output stayed aligned to a useful task boundary | An oversized task, unrelated tangents, or a mid-session objective change |
The dimension comment explains the model’s rating using evidence from the session. It can paraphrase or quote transcript content and follows the same access restrictions as the transcript.
What the report shows
Use the date, team or group, attribute, tool, contributor, aggregation, calculation, and comparison controls to define the cohort. Open a chart point or breakdown row to see evaluated sessions.
The session list can include contributor, title, AI tool, message count, Agent Experience Score, and date. Available detail depends on data sources and viewer permission.
Overview
The Overview tab can show the session summary, overall and dimension scores, evaluation comments, contributor, tool and version, message count, start time, and token usage when the integration provides it.
Output
The Output tab can link repositories, branches, commits, PRs, and the latest associated deployment. A session can have no linked output, and linked output can include work from other sessions or people.
Transcript
The Transcript tab shows scrubbed user and agent messages for authorized viewers. Tool integrations provide different combinations of timestamps, model data, and token counts.
Admins configure access to transcripts, titles, summaries, and evaluation comments under Admin → General settings → Access → AI Code Insights transcripts. See IC metrics.
Turn the score into action
- Choose one dimension and a cohort with enough evaluated sessions.
- Compare high- and low-scoring examples to confirm a repeated pattern.
- Inspect several transcripts and linked outputs where access permits.
- Change one workflow practice that maps to the evidence.
- Compare the same dimension, team, tool, and work type in the next period.
Examples of focused changes include a clearer issue template, smaller agent-sized tasks, updated AGENTS.md guidance, a reusable Skill, or a different review and testing sequence.
Use AI code percentage and AI pull request overview as downstream context. A change in those reports is an association, not proof that the Agent Experience intervention caused it.
Limitations
- The rubric measures the developer-agent interaction, not the correctness, security, maintainability, or business value of the resulting code.
- A model evaluation can be incomplete or wrong. Inspect representative sessions before acting on an aggregate.
- Not every session is captured or evaluated. Coverage depends on the machine, repository, tool integration, session lifecycle event, transcript setting, and evaluation eligibility.
- Tool and team comparisons can reflect different work types, codebases, task sizes, or session volumes.
- DX does not currently assign coaching actions or prescribe the correct intervention automatically.