In machine learning, evaluating model performance is crucial for understanding how well the model generalizes to new data. Different types of problems—classification, regression, or clustering—require different evaluation metrics.
1. Classification Metrics
Classification metrics are used to evaluate models that predict discrete labels (e.g., spam or not spam).
1.1 Accuracy
Accuracy is the ratio of correctly predicted instances to the total number of instances.
from sklearn.metrics import accuracy_score
y_true = [1, 0, 1, 1, 0, 1, 0]
y_pred = [1, 0, 1, 0, 0, 1, 1]
accuracy = accuracy_score(y_true, y_pred)
print("Accuracy:", accuracy)
1.2 Precision
Precision is the ratio of true positives to the total number of predicted positives. It answers the question: “Of all instances predicted as positive, how many are truly positive?”
from sklearn.metrics import precision_score
precision = precision_score(y_true, y_pred)
print("Precision:", precision)
1.3 Recall (Sensitivity)
Recall is the ratio of true positives to the total number of actual positives. It answers the question: “Of all actual positive instances, how many were correctly predicted?”
from sklearn.metrics import recall_score
recall = recall_score(y_true, y_pred)
print("Recall:", recall)
1.4 F1 Score
The F1 score is the harmonic mean of precision and recall. It provides a balance between the two metrics.
from sklearn.metrics import f1_score
f1 = f1_score(y_true, y_pred)
print("F1 Score:", f1)
1.5 ROC-AUC
ROC-AUC (Receiver Operating Characteristic – Area Under Curve) measures how well a model distinguishes between classes. A higher AUC indicates a better model.
from sklearn.metrics import roc_auc_score
y_prob = [0.9, 0.1, 0.8, 0.4, 0.2, 0.7, 0.3]
roc_auc = roc_auc_score(y_true, y_prob)
print("ROC-AUC:", roc_auc)
2. Regression Metrics
Regression metrics are used for evaluating models that predict continuous values (e.g., predicting house prices).
2.1 Mean Absolute Error (MAE)
MAE is the average of the absolute differences between predicted and actual values.
from sklearn.metrics import mean_absolute_error
y_true = [3.0, -0.5, 2.0, 7.0]
y_pred = [2.5, 0.0, 2.0, 8.0]
mae = mean_absolute_error(y_true, y_pred)
print("Mean Absolute Error:", mae)
2.2 Mean Squared Error (MSE)
MSE is the average of the squared differences between predicted and actual values. It penalizes larger errors more than smaller ones.
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(y_true, y_pred)
print("Mean Squared Error:", mse)
2.3 R-squared (Coefficient of Determination)
R-squared measures the proportion of variance explained by the model. It ranges from 0 to 1, where 1 indicates a perfect fit.
from sklearn.metrics import r2_score
r2 = r2_score(y_true, y_pred)
print("R-squared:", r2)
3. Clustering Metrics
Clustering metrics evaluate unsupervised learning models that group data into clusters.
3.1 Silhouette Score
The Silhouette Score measures how similar an instance is to its own cluster compared to other clusters.
from sklearn.metrics import silhouette_score
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
X, _ = make_blobs(n_samples=100, centers=3, random_state=42)
kmeans = KMeans(n_clusters=3)
kmeans.fit(X)
silhouette = silhouette_score(X, kmeans.labels_)
print("Silhouette Score:", silhouette)
4. Practical Tips for Choosing Evaluation Metrics
- For imbalanced datasets, prioritize metrics like Precision, Recall, and F1 Score.
- Use ROC-AUC for binary classification problems to assess the model’s overall performance.
- For regression problems, prefer MAE when minimizing errors is crucial and MSE for penalizing large errors.
- For clustering problems, evaluate multiple metrics like Silhouette Score and Inertia.
Conclusion
Evaluation metrics play a vital role in measuring the success of machine learning models. Choosing the right metric depends on the problem you are trying to solve.