Implement generative AI and agentic solutions
Tuning, Reflection, and Observability for Generative AI
ImportantTune prompts and generation parameters, choose and configure reasoning models, add reflection and self-critique loops, instrument tracing with token, safety, and latency signals, and orchestrate several models or combine models with deterministic rules.
Aligned to the AI-103 study guide with skills measured as of April 16, 2026; guide and product behavior verified October 12, 2026.
Why this matters
Once an app works, the job becomes making it cheaper, faster, more reliable, and easier to debug. AI-103 expects you to know which lever fixes which symptom: a parameter change, a reasoning setting, a verification loop, a routing decision, or a trace that shows where the time and tokens went.
Must Know
- Temperature controls randomness: lower values give focused, repeatable output and higher values give more varied output. Adjust temperature or top_p, not both.
- Prompt techniques for non-reasoning models include clear instructions, few-shot examples, delimiters, breaking tasks into steps, asking for citations, and giving the model an explicit way to say it cannot find an answer.
- Reasoning models spend hidden reasoning tokens that are billed as output tokens. Control depth with reasoning_effort, cap output with max_output_tokens or max_completion_tokens, and expect parameters such as temperature, top_p, and max_tokens to be unsupported on most reasoning models.
- Chain-of-thought prompting is for non-reasoning models; reasoning models already reason internally, and their reasoning summary parameter is the supported way to inspect a summary of that reasoning.
- Reflection and self-critique loops generate an answer, check it against criteria or sources (for example extracting claims and verifying each against retrieved passages, or scoring it with an evaluator), and revise or reject it before returning.
- End-to-end latency equals time to first token plus time between tokens multiplied by the number of generated tokens. Streaming improves perceived latency but not total generation time, and shorter outputs reduce total time.
- Foundry tracing uses OpenTelemetry. Spans capture model, tool, and retrieval steps with token usage and timing, and Application Insights dashboards show token consumption, latency, errors, and evaluation scores.
- Model router selects a model per request in Balanced, Quality, or Cost mode, and the response's model field shows which model served each request.
- Deterministic business rules (eligibility checks, thresholds, approvals) belong in code or workflow steps that run alongside model calls, not in prompts the model may ignore.
Compare and Distinguish
- Temperature versus top_p: scaling randomness of all tokens versus limiting sampling to the most probable token mass.
- Chain-of-thought prompting versus reasoning effort: asking a standard model to reason step by step versus setting how much a reasoning model thinks.
- Self-critique loop versus evaluation run: an inline check that changes the answer before it is returned versus offline scoring of many answers.
- Model router versus explicit multimodel orchestration: automatic per-prompt selection versus code or workflow that assigns each step to a chosen model.
- Time to first token versus time to last token: responsiveness versus total completion time.
Scenario examples
- Scenario: A contract-drafting assistant produces different wording every run, and lawyers want consistency. Think: lower temperature and keep top_p at its default.
- Scenario: A reasoning model returns empty answers with an incomplete status on long tasks. Think: raise max_output_tokens or lower reasoning_effort, because reasoning tokens consumed the budget.
- Scenario: A loan assistant must apply a fixed income-to-debt threshold exactly. Think: implement the threshold as a rules step in code and let the model explain the outcome.
Exam traps
- Setting temperature on most reasoning models is not supported; use reasoning_effort and output token limits instead.
- Streaming does not reduce the number of tokens generated or the total time to the last token.
- Adding chain-of-thought instructions to a reasoning model does not help and can conflict with its built-in reasoning.
- A reflection loop that critiques without sources can repeat the same unsupported claim; anchor the critique in retrieved evidence or explicit criteria.
- Trace message content is not captured by default; enabling it is a deliberate privacy decision.
Key takeaways
- Match the lever to the symptom: parameters for style, reasoning settings for depth, loops for verification.
- Measure latency and tokens per step with traces before optimizing.
- Keep exact rules in deterministic code and let models handle language and judgment.
How it works
- Each traced request produces a parent span with child spans for retrieval, model, and tool calls, so latency and token use can be attributed to a step.
- A verification step can call a second, often cheaper, model or an evaluator to accept, revise, or reject a draft answer.
Objects and administrative surfaces
- Model parameters in the Foundry playground and in Responses API requests.
- Application Insights transaction search, performance views, and the Foundry tracing view.
- Azure Monitor metrics such as Time to Response and Time to Last Byte.
- Agent Framework workflows that mix agent steps and function executors.
When to use it
- Use model router when traffic mixes simple and complex prompts and you want automatic cost and quality balancing.
- Use explicit routing in code when compliance requires a specific model for specific data.
Security and governance implications
- Redact personal data from spans and avoid recording secrets in trace attributes.
- Log which model served each request so audits can reproduce outcomes.
Troubleshooting signals
- High time to first token with normal time between tokens points to long prompts or queueing; long total time with normal first-token time points to long outputs.
- If costs jump after switching to a reasoning model, check reasoning token counts in usage details.
More detail
- Tune prompts, temperature, top_p, output limits, and reasoning effort.
- Implement reflection, chain-of-thought evaluation, and self-critique loops.
- Set up tracing with token, safety, and latency breakdowns.
- Orchestrate several models, flows, or hybrid model and rules engines.
Ready for the quiz?
- Why should you change temperature or top_p but not both?
- Which parameters control cost and depth for a reasoning model?
- What makes a self-critique loop more reliable than simply asking the model to double-check?
- Which trace or metric shows whether slowness comes from the model or from a tool?
Related objectives
- D2.3.S1 — Tune generation behavior, such as prompt engineering and adjusting model parameters
- D2.3.S2 — Implement model reflection, chain-of-thought evaluations, and self-critique loops
- D2.3.S3 — Set up observability by implementing tracing, token analytics, safety signals, and latency breakdowns
- D2.3.S4 — Orchestrate multiple models, flows, or hybrid LLM and rules engines