Testing, Validation, and Troubleshooting
Troubleshooting Prompts, Retrieval, and Integrations
Diagnose GenAI failures systematically across request schemas, context windows, prompt versions, embeddings, chunks, indexes, permissions, quotas, tools, and downstream dependencies.
Concepts
- Use correlation identifiers and stage-level traces to follow one request through retrieval, inference, tools, and validation.
- Low retrieval recall can result from chunking, embedding mismatch, stale indexes, metadata filters, or query formulation.
- Context truncation can silently remove instructions or evidence; record token counts and truncation decisions.
- Reproduce failures with versioned prompts, configuration, data snapshots, and sanitized request evidence.
Exam tips
- Check retrieval results before changing the model when answers lack source facts.
- Validate provider-specific payload schemas and quotas when an FM API call fails.
- Use CloudWatch Logs and distributed traces to distinguish model latency from tool or retrieval latency.