Fundamentals of AI and ML
Data Splits and Generalization
Data splitting helps estimate how well a model will perform on new data. Training, validation, and test sets have different jobs, and mixing them can create misleading performance results.
Concepts
- Training data is used to fit model parameters.
- Validation data is used during development for model selection and hyperparameter tuning.
- Test data is held out until final evaluation to estimate performance on unseen data.
- Cross-validation rotates validation folds to get a more stable estimate on limited datasets.
Exam tips
- Use validation data for model selection and tuning, not final reporting.
- Use a held-out test set for the final estimate of unseen performance.
- Cross-validation is useful when data is limited.