categorical
Fleiss' kappa
Measure categorical agreement across three or more raters.
Requirements
- three or more rater columns
- a shared category scale
- the same rated observations
Avoid when
- different observations per rater
- numeric measurements
- rater-specific bias is central
Formula
κ = (P̄ − P̄ₑ) / (1 − P̄ₑ)
Interpretation
Fleiss' kappa summarizes agreement beyond chance across all raters and observations.
Worked example
Measure agreement among four reviewers assigning the same diagnostic categories.
Open example datasets