The accuracy paradox is the
paradoxical
A paradox is a logically self-contradictory statement or a statement that runs contrary to one's expectation. It is a statement that, despite apparently valid reasoning from true premises, leads to a seemingly self-contradictory or a logically u ...
finding that
accuracy
Accuracy and precision are two measures of ''observational error''.
''Accuracy'' is how close a given set of measurements (observations or readings) are to their ''true value'', while ''precision'' is how close the measurements are to each other ...
is not a good metric for
predictive model
Predictive modelling uses statistics to predict outcomes. Most often the event one wants to predict is in the future, but predictive modelling can be applied to any type of unknown event, regardless of when it occurred. For example, predictive mod ...
s when
classifying in
predictive analytics
Predictive analytics encompasses a variety of statistical techniques from data mining, predictive modeling, and machine learning that analyze current and historical facts to make predictions about future or otherwise unknown events.
In business ...
. This is because a simple model may have a high level of accuracy but be too crude to be useful. For example, if the incidence of category A is dominant, being found in 99% of cases, then predicting that case is category A will have an accuracy of 99%.
Precision and recall are better measures in such cases.
[
The underlying issue is that there is a class imbalance between the positive class and the negative class.]
Example
For example, a city of 1 million people has ten terrorists. A profiling system results in the following confusion matrix
In the field of machine learning and specifically the problem of statistical classification, a confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm, typically a su ...
:
Even though the accuracy is ≈ 99.9%, 990 out of the 1000 positive predictions are incorrect. The precision of = 1% reveals its poor performance. As the classes are so unbalanced, a better metric is the F1 score
In statistical analysis of binary classification, the F-score or F-measure is a measure of a test's accuracy. It is calculated from the precision and recall of the test, where the precision is the number of true positive results divided by the ...
= ≈ 2% (the recall being = 1).
References
Statistical paradoxes
{{statistics-stub