Plan and manage an Azure AI solution
Responsible AI Guardrails, Evaluation, and Agent Governance
CoreApply guardrails to models and agents, measure quality and safety with evaluators and automated red teaming, keep an audit trail through tracing and provenance, and govern what agents may do with approvals, constraints, and tool-access controls.
Aligned to the AI-103 study guide with skills measured as of April 16, 2026; guide and product behavior verified October 12, 2026.
Why this matters
Generative and agentic systems can produce harmful content, follow injected instructions, invent facts, or take actions nobody approved. Responsible AI on AI-103 is practical engineering: which control detects which risk, where it intervenes, how you prove behavior with evaluations and traces, and how you keep a human in charge of risky actions.
Must Know
- A Foundry guardrail is a named collection of controls. Each control pairs a risk with intervention points and an action. Intervention points are user input, output, and, for agents (preview), tool call and tool response.
- Content risks (hate, sexual, self-harm, violence) use severity thresholds. A low threshold flags low, medium, and high severity content; a high threshold flags only the most severe content. Turning a category off requires Microsoft approval for modified guardrails.
- Prompt Shields detects user prompt attacks at user input and document (indirect) attacks hidden in documents, email, web pages, or tool results. Other controls cover protected material, PII (preview), groundedness (preview), and task adherence (preview).
- An agent's own guardrail fully overrides the guardrail of its model deployment; without one, the agent inherits the model deployment's guardrail.
- A filtered prompt returns an HTTP 400 error, and a filtered completion ends with finish_reason set to content_filter, so applications must handle both cases.
- Quality evaluators include groundedness, relevance, coherence, fluency, retrieval, and response completeness. Agent evaluators include intent resolution, task adherence, and tool call accuracy, and risk and safety evaluators score harmful content, indirect attacks, protected material, and code vulnerabilities.
- The AI Red Teaming Agent automates adversarial probing with PyRIT-based attack strategies and reports attack success rate, so you can compare safety before and after a mitigation.
- Tracing with OpenTelemetry stored in Application Insights records each model call, tool call, and agent step. Retrieval with citations and versioned agents, prompts, and policies provides provenance for auditing.
- MCP tools require approval by default: require_approval can be always, never, or a per-tool list, and the app answers each approval request before the tool runs. allowed_tools restricts which server tools an agent can see.
Compare and Distinguish
- Guardrail controls versus evaluators: runtime detection and blocking versus offline or continuous measurement of behavior.
- User prompt attack versus document attack: a malicious instruction typed by the user versus one embedded in content the model reads.
- Groundedness detection versus groundedness evaluator: a runtime check on a response versus a scored evaluation across a dataset.
- Approval required versus allowed_tools: pausing for human consent before a call versus never exposing a tool at all.
Scenario examples
- Scenario: An agent reads supplier emails that might contain hidden instructions telling it to forward invoices. Think: Prompt Shields document attack detection on tool responses, plus approval on the email-sending tool.
- Scenario: A team must prove a new prompt did not increase ungrounded answers before release. Think: run groundedness and relevance evaluators on the same dataset for both versions in the pipeline.
- Scenario: Auditors want to reconstruct which tools an agent called and why for a disputed refund. Think: agent tracing in Application Insights with retained conversation and tool-call spans.
Exam traps
- A guardrail on the model deployment does not protect an agent that has its own guardrail assigned; the agent's guardrail replaces it.
- Evaluations do not block anything at runtime; pair them with guardrails for enforcement.
- Setting require_approval to never on an MCP server removes the human checkpoint for every tool on that server.
- Groundedness and spotlighting controls in preview apply to models but are not yet supported for agents.
- Message content is not captured in traces unless content recording is explicitly enabled, which also raises privacy considerations.
Key takeaways
- Enforce with guardrails, measure with evaluators, probe with red teaming, and audit with traces.
- Place controls at the intervention point where the risk enters, including tool responses for agents.
- Keep humans in the loop for consequential or irreversible agent actions.
How it works
- Guardrail classifiers from Azure AI Content Safety score content at each configured intervention point and annotate or block it according to the control's action.
- Red teaming sends generated adversarial prompts to the target and uses safety evaluators to judge whether each attack succeeded.
Objects and administrative surfaces
- Foundry portal Guardrails page for controls, intervention points, and assignments to models and agents.
- Foundry portal Evaluation and Red teaming areas, and the azure-ai-evaluation SDK.
- Application Insights traces linked to the Foundry project.
- MCP tool definitions with require_approval and allowed_tools settings.
When to use it
- Use per-agent guardrails when an agent's tools expose risks the base model deployment never sees.
- Use human approval for tools that send messages, move money, or change records.
Security and governance implications
- Changing guardrails requires an elevated Foundry role, so developers cannot quietly weaken safety settings.
- Retain traces according to policy, and avoid capturing sensitive message content unless it is required.
Troubleshooting signals
- If legitimate prompts are blocked, review annotations to see which category and severity fired before changing thresholds.
- If an agent ignores a guardrail you set on its model, check whether a different guardrail is assigned to the agent itself.
More detail
- Configure guardrails, severity thresholds, Prompt Shields, and other risk controls.
- Run quality, agent, and risk and safety evaluations and automated red teaming.
- Implement auditing with tracing, citations, versioning, and approval records.
- Govern agents with approvals, allowed tool lists, instructions, and least-privilege identities.
Ready for the quiz?
- Which intervention point catches instructions hidden in a web page returned by a tool?
- What happens to a model deployment's guardrail when an agent has its own guardrail?
- Which evaluators would you use to compare two agent versions on correct tool use?
- How do require_approval and allowed_tools differ?
Related objectives
- D1.4.S1 — Configure safety filters, guardrails, risk detection, and content moderation
- D1.4.S2 — Apply responsible AI instrumentation, including evaluators, safety evaluations, and explanation tooling
- D1.4.S3 — Implement auditing through trace logging, provenance metadata, and approval workflows
- D1.4.S4 — Govern agent behavior with oversight modes, constraints, and tool-access controls