Abstract
Commonly used evaluation measures including Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic. Using these measures a system that performs worse in the objective sense of Informedness, can appear to perform better under any of these commonly used measures. We discuss several concepts and measures that reflect the probability that prediction is informed versus chance. Informedness and introduce Markedness as a dual measure for the probability that prediction is marked versus chance. Finally we demonstrate elegant connections between the concepts of Informedness, Markedness, Correlation and Significance as well as their intuitive relationships with Recall and Precision, and outline the extension from the dichotomous case to the general multi-class case.
Keywords
MarkednessMeasure (data warehouse)RecallComputer scienceStatisticCorrelationNatural language processingArtificial intelligenceClass (philosophy)Extension (predicate logic)StatisticsMathematicsPsychologyLinguisticsData miningCognitive psychology
Affiliated Institutions
Related Publications
Some Concepts of Dependence
Problems involving dependent pairs of variables $(X, Y)$ have been studied most intensively in the case of bivariate normal distributions and of $2 \\times 2$ tables. This is du...
THE ANOMALOUS BEHAVIOUR OF PRECISION IN THE SWETS MODEL, AND ITS RESOLUTION
M. H. Heine has shown that if one follows the retrieval procedure associated with Swets' model of an information retrieval system it is possible that the inverse relationship be...
MEASURES OF LANGUAGE EFFECTIVENESS AND THE SWETSIAN HYPOTHESES
‘Language measures’ such as Swets's E or Brookes's S, which measure the separation of the PMFs defined by a weighting formula applied to the sets of relevant and non‐relevant do...
THE CORRELATION‐BASED LAW OF EFFECT<sup>1</sup>
It is commonly understood that the interactions between an organism and its environment constitute a feedback system. This implies that instrumental behavior should be viewed as...
Testing Statistical Hypotheses
This chapter presents the basic concepts and results of the theory of testing statistical hypotheses. The generalized likelihood ratio tests that are discussed can be applied to...
Publication Info
- Year
- 2020
- Type
- preprint
- Volume
- 2
- Issue
- 1
- Pages
- 37-63
- Citations
- 4425
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
4425
OpenAlex
Cite This
David Powers
(2020).
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation.
arXiv (Cornell University)
, 2
(1)
, 37-63.
https://doi.org/10.48550/arxiv.2010.16061
Identifiers
- DOI
- 10.48550/arxiv.2010.16061