Implement computer vision solutions
Multimodal Visual Understanding and Content Understanding
ImportantUse multimodal models to describe, caption, and answer questions about images; produce accessible alt text; configure Azure Content Understanding analyzers for images and video; segment and summarize video; and locate objects or regions in visual content.
Aligned to the AI-103 study guide with skills measured as of April 16, 2026; guide and product behavior verified October 12, 2026.
Why this matters
Visual understanding powers accessibility, cataloging, inspection, and video search. AI-103 expects you to decide when a prompt to a multimodal model is enough and when a Content Understanding analyzer with a schema, segmentation, and grounding is the better pipeline, and to configure either for concise or detailed output.
Must Know
- Vision-capable chat models (for example GPT-4o, GPT-4.1, GPT-5 series, and o-series models) accept images as URLs or base64 data alongside text, up to 10 images per request, so one prompt can compare or caption several images together.
- The detail setting on an image input controls cost and fidelity: low uses a single low-resolution pass, high adds detailed tiles for small text and objects, and auto lets the model decide.
- Caption length and style come from instructions and output limits: ask for a one-sentence caption or a detailed description, and set a maximum output token limit so responses are not cut off.
- Grounded visual question answering means instructing the model to answer only from what is visible and to say when the image does not show the answer, rather than inferring from general knowledge.
- Alt text should be concise and convey the image's purpose in context, while an extended description covers details such as chart values or diagram structure for users who need more.
- Azure Content Understanding in Foundry Tools uses analyzers. Prebuilt analyzers such as prebuilt-imageSearch and prebuilt-videoSearch produce RAG-ready descriptions, and custom analyzers derive from base analyzers (prebuilt-image, prebuilt-video, prebuilt-document, prebuilt-audio) with a field schema.
- Field extraction methods are extract (documents only), classify (pick from defined categories), and generate (produce values such as descriptions). Content Understanding requires your own Foundry model deployments for generative features.
- For video, content extraction produces a WebVTT transcript, key frames, and shot boundaries, and segmentation (enableSegment with a category description) splits the video into logical segments that each receive generated fields.
- A standard analyzer applies one extraction task per file. Reasoning modes (pro mode in the 2025-05-01 preview API, and agentic mode for documents in the 2026-06-01 preview API) derive answers that require calculation or validation, and contentCategories routes segments to specialized subanalyzers.
- To identify objects or regions, define classify or generate fields (for example products present or defect type) in a custom image or video analyzer, or ask a multimodal model to locate items; Content Understanding samples video at about one frame per second, so very brief events can be missed.
Compare and Distinguish
- Multimodal prompt versus Content Understanding analyzer: flexible ad hoc answers versus a repeatable schema with consistent fields across files.
- Caption versus alt text versus extended description: short summary versus purpose-focused accessible text versus a full account of visual details.
- Whole-video analysis versus segmented analysis: one set of fields for the entire video versus fields per scene or topic segment.
- Standard analyzer versus reasoning mode: direct extraction or generation versus answers that must be derived or validated across evidence.
Scenario examples
- Scenario: A museum website needs short alt text for 50,000 artwork photos plus a longer description on request. Think: a vision model with instructions for a one-sentence alt text and a separate detailed-description mode.
- Scenario: A media company wants each news broadcast split into stories, each with a summary and a list of people shown. Think: a custom video analyzer with segmentation by story and generated fields.
- Scenario: A factory wants every product photo labeled scratch, dent, or no defect in a consistent field. Think: a custom image analyzer with a classify field listing the allowed values.
Exam traps
- Setting detail to low saves tokens but can miss small text and objects that the question depends on.
- The extract method applies only to documents; image and video fields use classify or generate.
- Segmentation consumes generative model tokens even when no fields are defined.
- Pro-mode reasoning was a preview feature; current reasoning over documents uses a preview agentic mode with one input file per request.
- An answer that is plausible but not visible in the image is a grounding failure, even if it happens to be true.
Key takeaways
- Use multimodal prompts for flexible visual Q&A and captions, and Content Understanding for repeatable schemas.
- Write alt text for purpose and extended descriptions for detail.
- Segment video into meaningful units before extracting fields from each.
How it works
- Content Understanding first extracts content such as transcripts and key frames, then a generative model fills the field schema for the file or for each segment.
- Analyze calls are asynchronous: the service returns an Operation-Location URL that you poll until the result succeeds.
Objects and administrative surfaces
- Content Understanding Studio and the Foundry portal for analyzers and test runs.
- Content Understanding REST API and SDKs with the analyze operation and Operation-Location polling.
- Chat Completions or Responses API with image_url content parts.
- Analyzer JSON with baseAnalyzerId, models, config, and fieldSchema.
When to use it
- Use prebuilt-videoSearch when you need RAG-ready transcripts and segment summaries without custom fields.
- Use a custom analyzer when downstream systems need named fields with fixed categories.
Security and governance implications
- Content Understanding inherits guardrails from the model deployments it uses, so filtered content can suppress generated fields.
- Image URLs sent to a model must be reachable by the service; private storage needs an access method that the service can use.
Troubleshooting signals
- If a video analyzer misses a brief on-screen event, remember the roughly one-frame-per-second sampling and consider higher-detail frame analysis for that clip.
- If captions are truncated, raise the output token limit or ask for a shorter style.
More detail
- Analyze visual context and answer grounded questions with multimodal models.
- Produce concise or detailed captions, alt text, and extended descriptions.
- Configure Content Understanding image and video analyzers, segmentation, and routing.
- Identify objects, components, or regions in images and video.
Ready for the quiz?
- When does high image detail justify its extra tokens?
- Which field methods can an image analyzer use?
- How does segmentation change the output of a video analyzer?
- What distinguishes alt text from an extended image description?
Related objectives
- D3.2.S1 — Build a solution that analyzes visual context by using multimodal models
- D3.2.S2 — Configure apps to produce concise or detailed captions for single or multiple images
- D3.2.S3 — Implement a solution that enables question‑answering grounded in visual evidence
- D3.2.S4 — Configure generation of alt‑text and extended image descriptions aligned to accessibility guidelines
- D3.2.S5 — Implement visual understanding by configuring Azure Content Understanding in Foundry Tools to extract visual characteristics
- D3.2.S6 — Implement video analysis workflows to process and interpret video segments
- D3.2.S7 — Configure single‑task and pro‑mode Content Understanding pipelines
- D3.2.S8 — Implement solutions that identify objects, components, or regions within images or video