How does the Agent Experience Score work?
The Agent Experience Score evaluates how clearly eligible coding-agent sessions were framed, steered, and scoped. Use it to find recurring interaction friction, inspect representative sessions, and choose a specific workflow improvement.
Note: This feature requires AI Code Insights and Transcripts & Agent Experience enabled by a DX admin. Transcript access for each viewer is controlled separately.
When to use Agent Experience Score
Agent Experience Score answers the question: Where do captured coding-agent sessions repeatedly encounter interaction friction?
This data helps teams:
- Find a recurring failure mode — Compare requirements, steering, and scope across teams, contributors, attributes, and tools.
- Inspect the evidence — Open the sessions and evaluation comments behind a score.
- Target one intervention — Improve task context, issue templates, agent guidance, Skills, configuration, or task sizing.
- Remeasure the same cohort — Check whether the pattern changes after a documented intervention.
Do not use the score as a code-quality grade, employee ranking, developer-satisfaction measure, or ROI calculation.
How Agent Experience Score is calculated
After an eligible session ends, DX evaluates its scrubbed transcript with a separate model. The coding agent used during the session does not self-report the score.
Under the current rubric, an evaluated session receives a 1–5 rating for three dimensions. Its session score is:
(Requirements + Steering + Scope) ÷ 3
Older stored sessions can include legacy evaluation dimensions. Sessions with insufficient evidence, an unreadable transcript, or unsupported length can remain unevaluated. Missing scores should not be treated as low or high scores.
Scoring dimensions
| Dimension | What the evaluation considers | A low score can point to |
|---|---|---|
| Requirements | Whether the goal was clear and the initial context helped the agent begin | Missing acceptance criteria, unclear goals, or insufficient repository and domain context |
| Steering | Whether follow-up input helped guide progress | Ambiguous corrections, delayed redirection, or feedback that did not resolve the blocker |
| Scope | Whether the output stayed aligned to a useful task boundary | An oversized task, unrelated tangents, or a mid-session objective change |
The dimension comment explains the model’s rating using evidence from the session. It can paraphrase or quote transcript content and follows the same access restrictions as the transcript.
Turn the score into action
- Choose one dimension and a cohort with enough evaluated sessions.
- Compare high- and low-scoring examples to confirm a repeated pattern.
- Inspect several transcripts and linked outputs where access permits.
- Change one workflow practice that maps to the evidence.
- Compare the same dimension, team, tool, and work type in the next period.
Examples of focused changes include a clearer issue template, smaller agent-sized tasks, updated AGENTS.md guidance, a reusable Skill, or a different review and testing sequence.
Use AI code percentage and AI pull request overview as downstream context. A change in those reports is an association, not proof that the Agent Experience intervention caused it.
Limitations
- The rubric measures the developer-agent interaction, not the correctness, security, maintainability, or business value of the resulting code.
- A model evaluation can be incomplete or wrong. Inspect representative sessions before acting on an aggregate.
- Not every session is captured or evaluated. Coverage depends on the machine, repository, tool integration, session lifecycle event, transcript setting, and evaluation eligibility.
- Tool and team comparisons can reflect different work types, codebases, task sizes, or session volumes.
- DX does not currently assign coaching actions or prescribe the correct intervention automatically.