Fundamentals of AI and ML
Traditional ML Metrics
Traditional metrics depend on the problem type. Classification, regression, and ranking tasks use different measures, and the best metric should match the business cost of errors.
Concepts
- Accuracy measures overall correctness but can hide class imbalance problems.
- Precision answers: when the model predicted positive, how often was it right?
- Recall answers: of actual positives, how many did the model catch?
- F1 is the harmonic mean of precision and recall.
- AUC-ROC summarizes ranking quality across classification thresholds.
- MSE, RMSE, MAE, and R-squared are common regression metrics.
- The same recall and precision idea reappears in RAG evaluation as context recall and context precision, applied to retrieved chunks instead of predictions.
Exam tips
- Precision focuses on correctness of positive predictions; recall focuses on catching actual positives.
- F1 balances precision and recall.
- RMSE, MAE, MSE, and R-squared are regression metrics.