Inter-annotator agreement
Inter-annotator agreement (IAA) measures how consistently different annotators assign the same labels to the same items, showing whether a labeling task is well defined rather than whether the labels are correct.[1] It is typically quantified with chance-corrected coefficients (Cohen’s kappa for two raters, Fleiss’ kappa for several, and Krippendorff’s alpha for missing data or non-nominal scales)[4], with no single score universally acceptable as “good enough” in data-labeling workflows.[10]
Purpose
High agreement suggests a well-defined task whose annotations can be trusted, while low agreement often points to ambiguity in the task or weak annotation guidelines.[1] Artstein and Poesio state the underlying premise: annotation data is reliable only if annotators agree, and if they do not, “either some of them are wrong or else the annotation scheme is inappropriate for the data”.[2]
Common metrics
Raw percent agreement is the simplest measure; McHugh names it as one of several measures used for reliability.[6]
Cohen’s kappa
Cohen’s kappa, introduced by Jacob Cohen in 1960, measures agreement between two raters on nominal categories and corrects for chance agreement.[3]
Fleiss’ kappa
Fleiss’ kappa extends the approach to more than two raters. It is commonly described as an extension of Cohen’s kappa, although one study notes that it actually generalises Scott’s pi.[4] It cannot handle missing data except by excluding every observation that has a missing value.[4]
Krippendorff’s alpha
Krippendorff’s alpha supports any number of raters, several measurement levels (nominal, ordinal, interval) and missing data.[4] It is based on observed disagreement corrected for the disagreement expected by chance.[4][5] Artstein and Poesio discuss alpha alongside Scott’s pi and Cohen’s kappa, and argue that alpha-style coefficients, less used in computational linguistics than kappa-style ones, may suit many annotation tasks better, though their values are harder to interpret.[2]
Other measures
McHugh also lists the intraclass correlation coefficient and Pearson and Spearman correlations among measures used for reliability.[6]
Choosing a metric
For two raters with complete nominal data, alpha closely approximates Cohen’s kappa, and with multiple raters it parallels Fleiss’ kappa.[4] A simulation study on complete nominal data found Fleiss’ kappa and Krippendorff’s alpha, each with bootstrap confidence intervals, equally suitable. For incomplete data or non-nominal scales it recommended alpha, and it advised against the asymptotic confidence interval for Fleiss’ kappa.[4] Some practitioner sources nonetheless treat alpha as the default choice.[9]
Interpreting scores
Landis and Koch (1977) proposed widely cited bands for kappa:[7]
| Kappa | Label |
|---|---|
| Below 0 | Poor |
| 0.00–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
Landis and Koch acknowledged that these benchmarks are subjective and offered them as a guideline.[8] Fleiss later proposed a scale that collapses the lowest three ranges into “0.40 or less”, labelled poor.[8] McHugh argues that Cohen’s suggested interpretation may be too lenient for health research, since it implies a kappa as low as 0.41 might be acceptable.[6] For Krippendorff’s alpha, practitioner sources commonly cite values above 0.667 as tentatively acceptable and above 0.80 as reliable.[9]
CASRAI notes that no single value is universally acceptable, because kappa depends on more than coder accuracy.[10] Comparing a coefficient with published benchmarks has also reportedly tended to give an overly optimistic picture of agreement.[8]
See also
References
See the references frontmatter.