Fundamentals of Generative AI
Foundation Models and Transformers
Foundation models are large pre-trained models that can be adapted to many tasks. Transformers, tokens, and context windows are core concepts behind many language and multimodal models tested on AIF-C01.
Concepts
- Foundation models are large pre-trained models adaptable to many downstream tasks.
- Transformers use attention mechanisms to model relationships between tokens in context.
- Tokens are the text units processed by language models; they may be words, subwords, characters, or symbols.
- Context windows limit how many tokens a model can consider at one time.
Exam tips
- A context window limits how much input and conversation history a model can consider.
- Tokens are model-processing units, not always full words.
- Foundation models can support many downstream tasks with prompting, RAG, or customization.