In the world of machine learning, selecting the right evaluation metric is critical to building trustworthy models—especially when dealing with imbalanced datasets, such as fraud detection, medical diagnosis, or spam classification. While Accuracy, F1 Score, ROC AUC, and PR AUC are commonly used, each has its strengths and ideal use cases.

So, how do you choose the right one? Let’s break it down.

🔹 1. Accuracy – The Simplest Metric

Definition:
Accuracy = (TP + TN) / (TP + TN + FP + FN)
It tells you what percentage of total predictions were correct.

Use When:

• Your classes are balanced (i.e., both positive and negative examples are roughly equal)
• The cost of false positives and false negatives is similar

Why It Can Be Misleading:
In highly imbalanced datasets, a model can achieve high accuracy simply by predicting the majority class. For instance, in a dataset with 95% negative samples, predicting all negatives gives 95% accuracy—but 0% usefulness.

🔹 2. F1 Score – The Balance Between Precision and Recall

Definition:
F1 Score = 2 * (Precision * Recall) / (Precision + Recall)

• Precision = TP / (TP + FP)
• Recall = TP / (TP + FN)

Use When:

• You care equally about false positives and false negatives
• You're dealing with imbalanced datasets
• You want a single score that captures both precision and recall trade-offs

Why It’s Powerful:
F1 is especially useful when false positives and false negatives are both costly, such as in fraud detection or medical diagnoses. It ensures that the model doesn't optimize one at the expense of the other.

🔹 3. ROC AUC – How Well Your Model Separates Classes

Definition:
ROC AUC (Receiver Operating Characteristic - Area Under Curve) measures the ability of the model to distinguish between classes across different thresholds.

Use When:

• You want a threshold-independent evaluation
• The dataset is balanced or moderately imbalanced
• You’re comparing multiple classifiers

Why It Can Be Deceptive:
In heavily imbalanced datasets, ROC AUC may give overly optimistic results because it considers the true negative rate, which may dominate the evaluation.

🔹 4. PR AUC – Precision-Recall Area Under Curve

Definition:
PR AUC focuses on the trade-off between precision and recall at various thresholds.

Use When:

• You have imbalanced data
• You care more about correctly predicting the positive class
• You want a more realistic measure of performance for rare event detection

Why It’s Ideal for Imbalanced Data:
Unlike ROC AUC, PR AUC focuses on positive class prediction quality, which is what often matters most in domains like fraud detection, disease detection, or recommendation engines.

So, Which Metric Should You Choose?

And what's the conclusion?

• Accuracy is simple but misleading on imbalanced datasets.
• F1 Score balances false positives and negatives, ideal when both errors are costly.
• ROC AUC is great for ranking ability but can be deceptive when classes are imbalanced.
• PR AUC is the gold standard when your focus is on correctly identifying the minority class.