Fundamentals of Generative AI
Inference Parameters
Inference parameters control generation behavior, length, and cost. The exam often asks which setting makes outputs more deterministic, more varied, shorter, or stopped at a boundary.
Concepts
- Temperature controls randomness; lower values are more deterministic, higher values are more varied.
- Top-p nucleus sampling limits candidates to a cumulative probability mass.
- Top-k limits sampling to the k most likely next tokens where supported.
- Max tokens limits response length and cost.
- Stop sequences tell a model when to stop generating.
Exam tips
- Lower temperature is more deterministic; higher temperature is more varied.
- Max tokens limits generated length and can control cost.
- Stop sequences end generation when a specified pattern appears.