Testing, Validation, and Troubleshooting
GenAI Evaluation and Deployment Quality Gates
Evaluate model, prompt, RAG, agent, safety, latency, and business behavior with representative datasets, automated metrics, human review, controlled experiments, and regression gates.
Concepts
- Use task-specific rubrics for correctness, relevance, faithfulness, safety, style, latency, cost, and completion.
- Evaluate retrieval and generation separately so a grounded answer failure can be localized.
- LLM-as-a-judge scales evaluation but must be calibrated against human labels and checked for bias.
- Canary and A/B tests require guardrail metrics and rollback thresholds in addition to engagement measures.
- Use Bedrock RAG evaluations for retrieve-only or retrieve-and-generate evidence, and AgentCore Evaluations for task, tool, and trajectory behavior from instrumented agent traces.
Exam tips
- Bedrock Model Evaluation supports automatic and human evaluation workflows.
- A fixed regression set makes prompt and model version comparisons reproducible.
- Agent evaluation includes tool selection, argument validity, task completion, steps, cost, and safety.