Implement generative AI and agentic solutions
Generative Applications with Foundry Models and RAG
CoreDeploy and call language, small, code, and multimodal models through a Foundry project, ground answers with retrieval-augmented generation, design tool-augmented and multistep flows, evaluate quality and safety, and connect applications with the Foundry SDK and OpenAI-compatible clients.
Aligned to the AI-103 study guide with skills measured as of April 16, 2026; guide and product behavior verified October 12, 2026.
Why this matters
Most AI-103 solutions are applications that call a model, retrieve grounding data, and return a checked answer. If the client points at the wrong endpoint, retrieval returns weak chunks, or nobody measures groundedness, the app ships fabrications with confidence. This topic covers the everyday developer path from deployment to evaluated, connected application.
Must Know
- A Foundry project endpoint has the form https://<resource-name>.services.ai.azure.com/api/projects/<project-name>. The Microsoft Foundry SDK project client (azure-ai-projects 2.x) manages connections, agents, and evaluations, and get_openai_client returns an OpenAI-compatible client for responses on the same project.
- The v1 Azure OpenAI API uses the standard OpenAI() client with a base URL ending in /openai/v1/, so you no longer pass a monthly api-version. For Entra ID authentication, pass a bearer token provider in place of the API key.
- The Responses API is the recommended way to call models and agents. Chat Completions remains available, and both accept text and, for vision-capable models, image inputs.
- Retrieval-augmented generation retrieves relevant chunks at query time and adds them to the prompt so the model answers from current, private data without retraining. Hybrid search with semantic ranking is the classic Azure AI Search pattern, and agentic retrieval decomposes complex questions into parallel subqueries.
- Structured outputs enforce a JSON schema when strict is true: every property must be listed as required, additionalProperties must be false, and optional fields are modeled as a type union with null. JSON mode only guarantees syntactically valid JSON.
- In function calling, the model proposes a function name and JSON arguments, your code runs the function, and you send the result back in a follow-up request so the model can finish the answer.
- Groundedness measures whether a response stays within the provided context (fabrication), relevance measures whether it addresses the query, coherence and fluency measure writing quality, and risk and safety evaluators score harmful content.
- Connections in a Foundry project store references and credentials for resources such as Azure AI Search or Storage, so application code can look them up by name instead of embedding endpoints and keys.
- Use embeddings through the OpenAI-compatible /openai/v1 endpoint; the project endpoint does not currently route embedding requests.
Compare and Distinguish
- Project client versus OpenAI-compatible client: Foundry-native operations such as connections and agent versions versus model responses, conversations, and evaluations.
- Structured outputs versus JSON mode: schema adherence versus only valid JSON.
- Groundedness versus relevance: answer supported by the retrieved context versus answer on-topic for the question.
- Classic RAG versus agentic retrieval: one hybrid query per question versus planned parallel subqueries with merged references.
- Prompt-based extraction versus function calling: the model returns data in text versus the model asks your code to perform an action.
Scenario examples
- Scenario: A policy assistant must answer only from the latest HR handbook stored in Blob Storage. Think: index the handbook in Azure AI Search and implement RAG with citations rather than fine-tuning.
- Scenario: A downstream system rejects any response missing required JSON fields. Think: structured outputs with strict set to true and a complete required list.
- Scenario: A team must catch answers that add facts not present in retrieved passages. Think: the groundedness evaluator on a test set that includes the retrieved context.
Exam traps
- Fine-tuning does not keep an assistant current with documents that change weekly; retrieval does.
- JSON mode does not guarantee that required fields are present.
- The model never executes a function itself; if the app does not run it and return the output, the conversation stalls.
- A relevant answer can still be ungrounded, so relevance alone does not detect fabrication.
- Hard-coding a search admin key in the app bypasses the project connection and identity model.
Key takeaways
- Call models through the project endpoint or the v1 OpenAI-compatible endpoint with Entra ID tokens.
- Ground answers with retrieval and enforce output shape with structured outputs.
- Evaluate groundedness, relevance, and safety before and after every change.
How it works
- At query time the app embeds or rewrites the question, retrieves top chunks, and sends them with instructions that require answering only from the supplied sources.
- Evaluators use a judge model or a hosted safety service to score each test row, and results roll up into pass rates per metric.
Objects and administrative surfaces
- Foundry portal playgrounds with View code for Python samples.
- Python packages azure-ai-projects, azure-identity, and openai.
- Project connections and the project endpoint shown on the project overview page.
- Foundry Evaluation pages and the azure-ai-evaluation evaluators.
When to use it
- Use small or code-specialized models for constrained tasks such as classification or code completion at scale.
- Use multimodal models when the input includes images, charts, or screenshots.
Security and governance implications
- Use DefaultAzureCredential and a data-plane role instead of API keys in application code.
- Store downstream resource credentials in project connections, not in source control.
Troubleshooting signals
- A 404 from the client often means the base URL omits /openai/v1/ or names the wrong deployment.
- If RAG answers miss obvious facts, inspect the retrieved chunks before changing the prompt.
More detail
- Deploy and call LLMs, SLMs, code models, and multimodal models.
- Implement RAG with Azure AI Search, hybrid queries, and citations.
- Design tool-augmented and multistep flows with function calling and structured outputs.
- Evaluate fabrication, relevance, quality, and safety, and connect apps to a Foundry project.
Ready for the quiz?
- What does the project client provide that the OpenAI-compatible client does not?
- Which schema rules does strict structured output require?
- Which evaluator detects an answer that adds unsupported facts?
- What must your application do after the model returns a function call?
Related objectives
- D2.1.S1 — Deploy and consume LLMs, small models, code models, and multimodal models
- D2.1.S2 — Implement retrieval-augmented generation (RAG) in an application
- D2.1.S3 — Design workflows, tool-augmented flows, and multistep reasoning pipelines
- D2.1.S4 — Evaluate models and apps, including detecting fabrications, relevance, quality, and safety
- D2.1.S5 — Integrate generative workflows into applications by using Foundry SDKs and connectors
- D2.1.S6 — Configure an application to connect to a Foundry project