Table of contents
Throughout this series, we’ve explored various ML models and their evaluation techniques. In this blog, let’s summarize key metrics.
Evaluating Machine Learning Models
In the world of machine learning, we want to teach computers to make predictions, like whether a student will pass or fail a test. To see how well our computer model is doing, we need to evaluate its performance using some key techniques and metrics.
Classification Metrics
Imagine you have a basket of fruits, and you want to teach a friend to identify apples. You show them many examples, and then you test them with some new fruits they haven't seen before. This is similar to the train-test-split technique. You use most of the fruits to train your friend (the training set 70-80%) and keep a few to test their knowledge (the test set 20-30%).
Now, when your friend makes predictions, you want to know how accurate they are. This is where metrics come in:
confusion matrix
A table showing true vs. predicted labels, highlighting true positives, true negatives, false positives, and false negatives.
It is like a scoreboard that shows how many apples were correctly or incorrectly identified. It helps you see where your friend made mistakes.
| predicted TRUE | predicted FALSE | ||
| actual TRUE | TRUE POSITIVE (TP) | FALSE NEGATIVE (FN) | P = TP+FN |
| actual FALSE | FALSE POSITIVE (FP) | TRUE NEGATIVE(TP) | N = FP+TP |
| P’ = TP+FP | N’ =FN+TP | TOTAL |
Accuracy
Ratio of correctly predicted instances to total instances.
It tells you how many apples your friend identified correctly out of all the fruits they guessed.
Accuracy = (TP+TN)/ TOTAL
Error Rate = 1-Accuracy = (FP+FN)/TOTAL
Precision (Exactness)
The fraction of true positives among predicted positives.
measures how many of the fruits your friend said were apples that were actually apples.
Precision = TP / P’
Recall (Completeness/ Sensitivity)
The fraction of true positives among actual positives.
It at how many actual apples your friend identified out of all the apples in the basket.
Recall = TP / P
F1 score
It combines precision and recall to give you a single score that reflects how well your friend is doing overall.
The harmonic mean of precision and recall, useful when both metrics are important.
*F1-Score = 2 Precision Recall / (Precision + Recall)*
Why we can't simply use Accuracy every time?
While accuracy is a useful metric, it doesn't always provide a complete picture of a model's performance. Here are a few reasons why relying solely on accuracy can be misleading:
Imbalanced Datasets: If one class is much more common than another, a model can achieve high accuracy by simply predicting the majority class. For example, if 90% of your data is "fail" and 10% is "pass," a model that predicts "fail" for every instance would have 90% accuracy, but it wouldn't be useful.
False Positives and False Negatives: Accuracy doesn't differentiate between types of errors. In some situations, false positives (predicting a positive when it's actually negative) and false negatives (predicting a negative when it's actually positive) can have different consequences. For example, in medical diagnoses, missing a disease (false negative) can be more critical than incorrectly diagnosing it (false positive).
Precision and Recall: These metrics provide more insight into the model's performance, especially in cases where the cost of different types of errors varies. Precision tells you how many of the predicted positives were actually positive, while recall tells you how many actual positives were correctly predicted.
Using a combination of metrics, including accuracy, precision, recall, and F1 score, gives a more comprehensive view of how well a model is performing.
Regression Metrics
Understanding how accurately a model predicts continuous numerical values.
Error Measurement: The difference between actual data points and the regression line, known as errors.
MAE (Mean Absolute Error): Average absolute difference between predicted and actual values.
Variance: Sum of squared differences between predictions and the average of actual target data.
MSE (Mean Squared Error): Sum of squared differences divided by the number of data points.
RMSE (Root Mean Squared Error): Square root of MSE, easier to interpret as it shares units with the target variable.
R-squared: Proportion of variance in the dependent variable explained by the independent variable, ranging from 0 (poor fit) to 1 (perfect fit).
Visualize R square
case 1: if model fits well. MSE of regression line is nearly zero
Case 2: if model doesn’t fit well
Evaluating Unsupervised Learning Models: Heuristics and Techniques
Unsupervised Learning: This involves discovering hidden patterns in data without predefined labels. It includes techniques like clustering and dimensionality reduction.
Evaluation Challenges: Unlike supervised learning, there are no clear answers to guide the evaluation of unsupervised models. Therefore, we assess the quality of patterns and groupings.
Internal Metrics: Evaluate clustering quality based on input data (e.g., silhouette score, Davies-Bouldin index).
External Metrics: Use ground truth labels to compare clustering results (e.g., adjusted Rand index, normalized mutual information).
Dimensionality Reduction Evaluation: Assess how well reduced data retains important information (e.g., explained variance ratio, reconstruction error).
Stability and Generalizability: It's important to ensure that the model produces consistent results across different data subsets.
Combination of Methods: Effective evaluation often requires a mix of heuristics, metrics, domain expertise, and visualization tools.
Ensuring Model Generalizability
Cross-Validation and Advanced Model Validation Techniques
Model Validation:
Model validation is the process of evaluating a machine learning model to ensure it performs well on unseen data. It involves assessing the model's ability to generalize beyond the training dataset, which is crucial for making reliable predictions in real-world applications.
Data Snooping:
Data snooping refers to the practice of checking a model's performance on the test data during the model optimization process. This can lead to overfitting, where the model learns patterns specific to the test data rather than generalizing well to unseen data.
Key Points about Data Snooping:
Leads to Misleading Results: If you evaluate your model on the test data while tuning it, you may mistakenly believe it performs better than it actually does on new data.
Data Leakage: It is a form of data leakage, where information from the test set influences the model training, compromising its validity.
Best Practices: To avoid data snooping, you should separate your data into training, validation, and test sets, using the test set only for final evaluation after model tuning
Key Aspects of Model Validation:
Training, Validation, and Test Sets: The data is typically divided into three parts:
Training Set: Used to train the model.
Validation Set: Used to tune hyperparameters and optimize the model.
Test Set: Held back for final evaluation to assess the model's performance on unseen data.
Avoiding Overfitting: Model validation helps prevent overfitting by ensuring that the model does not just memorize the training data but can also predict accurately on new data.
Cross-validation
Cross-validation is a statistical method used to assess how well a machine learning model generalizes to an independent dataset. It helps in evaluating the model's performance and ensuring that it does not overfit the training data.
K-Fold Cross-Validation:
The dataset is divided into K equal-sized folds.
For each fold, the model is trained on K-1 folds and tested on the remaining fold.
This process is repeated K times, with each fold serving as the test set once.
The final performance metric is averaged over all K trials, providing a more reliable estimate of model performance.
Benefits:
Better Utilization of Data: Every data point is used for both training and validation, maximizing the use of available data.
Reduces Overfitting: By testing the model on different subsets, it smooths out the evaluation and reduces the risk of overfitting to a specific training set.
Improves Generalization: It provides a clearer picture of how the model will perform on unseen data.
Stratified Cross-Validation:
Stratified Cross-Validation is a variation of cross-validation that ensures the class distribution is preserved in each fold of the dataset. This technique is particularly useful when dealing with imbalanced datasets, where some classes have significantly more samples than others
Preserves Class Distribution: Each fold contains approximately the same proportion of each class as the entire dataset. This helps ensure that the model is trained and validated on a representative sample of the data.
Improves Model Evaluation: By maintaining the class distribution, stratified cross-validation provides a more accurate assessment of the model's performance, especially in classification tasks where class imbalance can skew results.
Reduces Variance in Performance Estimates: It helps in obtaining more stable and reliable performance metrics, as each fold reflects the overall dataset's characteristics.
Regularization in Regression and Classification
What is Regularization?
Regularization is a technique used in machine learning to prevent overfitting, which occurs when a model learns the noise in the training data rather than the underlying pattern.
It achieves this by adding a penalty term to the cost function, which discourages overly complex models.
Cost Function
The standard cost function for linear regression is the Mean Squared Error (MSE), which measures the average squared difference between predicted and actual values.
With regularization, the cost function is modified to include a penalty term:
Regularized Cost Function = MSE + λ * Penalty Term
Here, λ (lambda) is a hyperparameter that controls the strength of the penalty.
Types of Regularization
Ridge Regression (L2 Regularization)
Adds a penalty equal to the sum of the squares of the coefficients:
- Penalty Term = Σ(θ_i^2)
This helps to shrink the coefficients but does not set any to zero, meaning all features are retained.
It is effective when dealing with multicollinearity (high correlation between features).
Lasso Regression (L1 Regularization)
Adds a penalty equal to the sum of the absolute values of the coefficients:
- Penalty Term = Σ|θ_i|
This can shrink some coefficients to exactly zero, effectively performing feature selection.
It is particularly useful when you suspect that many features are irrelevant.
Impact of Regularization
Bias-Variance Tradeoff: Regularization increases bias (by simplifying the model) but reduces variance (by making the model less sensitive to fluctuations in the training data).
Choosing λ: The value of λ is crucial. A small λ may lead to overfitting, while a large λ may lead to underfitting. Techniques like cross-validation can help in selecting the optimal λ.