Test and manage agents
Agent Evaluation and Test Sets
CoreCreate representative test sets, select evaluation methods, and interpret results before release.
Aligned to the AB-620 live Skills measured outline; visible page date April 21, 2026; product behavior verified September 19, 2026.
Why this matters
A few successful conversations do not establish reliability. Repeatable test sets expose regressions across intent selection, grounding, tool execution, response quality, and safety.
Must Know
- Create a test set: Build a versioned, representative set of prompts and expected criteria covering common, boundary, adversarial, no-result, and failure cases.
- Choose an evaluation method: Match deterministic, semantic, model-based, or human evaluation to the behavior and risk being measured.
- Review test results: Compare results by agent version, dataset, method, category, and failure pattern, then inspect traces before changing the design.
- Compare evaluation runs only with their agent version, dataset, method, and rubric in view; an aggregate score alone does not identify the failing behavior.
Compare and Distinguish
- A test set is reusable input and expected behavior; an evaluation method defines how responses are assessed; results identify patterns that require investigation.
- Deterministic checks fit exact routing or tool outcomes; model-based or rubric evaluation fits semantic response quality and still needs calibrated human review.
- Testing before release detects defects; production analytics and telemetry reveal real-world behavior after release.
Scenario examples
- Scenario: A pilot succeeds but production users have different permissions. Think: trace identity and authorization through every source, tool, and delegated agent.
- Scenario: A component works in development but fails after deployment. Think: inspect solution dependencies, connections, environment configuration, and channel-specific behavior.
- Scenario: A response is fluent but the operational result is uncertain. Think: separate generation from retrieval, tool execution, telemetry, and verification.
Exam traps
- Building a test set only from happy-path maker prompts.
- Comparing aggregate scores without checking whether the same dataset, version, and evaluation method were used.
- Treating an evaluator score as proof when the rubric is not aligned with the business requirement.
Key takeaways
- Reliable evaluation depends on representative reusable inputs, a method matched to the behavior under test, and result review at both aggregate and trace levels.
- Make the identity, data, lifecycle, and failure boundaries explicit for evaluate agent performance.
- Validate the behavior with representative evidence before broad deployment.
How it works
- Copilot Studio uses the configured descriptions, instructions, conversation state, and orchestration mode to select the relevant topic, knowledge, tool, flow, or connected agent.
- Runtime identity and environment configuration determine whether the selected capability can access its dependency and return a governed result.
Objects and administrative surfaces
- Copilot Studio evaluation pages for test sets, methods, runs, and results.
- Test pane for individual traces and activity maps.
- Analytics and Application Insights for post-publication evidence.
When to use it
- Use this topic when the requirement maps to evaluate agent performance.
- Prefer the narrowest supported surface that satisfies the required behavior and governance boundary.
Security and governance implications
- Apply least privilege to maker, runtime, connector, source, and deployment identities.
- Classify data and connectors, preserve audit evidence, and avoid exposing secrets or sensitive payloads in prompts and telemetry.
Troubleshooting signals
- Reproduce with the same version, channel, identity, inputs, and environment before changing configuration.
- Use activity traces, flow runs, tool errors, evaluation results, or telemetry to locate the failing layer.
More detail
- Build a reusable, versioned test set that includes routine prompts, edge conditions, hostile inputs, empty retrieval, and expected failure paths, with criteria drawn from the business requirement rather than only maker examples.
- Choose an evaluation method by the behavior being measured: exact assertions fit a required routing choice or operation result, while semantic quality needs a calibrated rubric, model-based assessment, or human review.
- Compare runs only when the agent version, dataset, and evaluation method are understood, then inspect categories, failure patterns, and individual traces before changing the design.
- Pre-release evaluation detects repeatable defects; post-publication analytics and telemetry reveal production behavior. Neither a single successful conversation nor an aggregate score proves reliability.
Ready for the quiz?
- Which requirement determines the correct surface for evaluate agent performance?
- Which identity performs the operation, and where is authorization enforced?
- What evidence distinguishes a safe successful result from a fluent but incomplete one?
Related objectives
- D3.1.S1 — Create a test set
- D3.1.S2 — Choose an evaluation method
- D3.1.S3 — Review test results