Interrater Agreement
Cross-source consensus on Interrater Agreement from 1 sources and 5 claims.
1 sources · 5 claims
How it works
Comparisons
Evidence quality
Highlighted claims
- Interrater agreement across all 14 appraisers averaged 0.59 (95% CI 0.48–0.70), spanning slight to substantial, while intrapair agreement averaged 0.75 (95% CI 0.68–0.82), spanning fair to almost perfect. — Agreement testing of AMSTAR-PF, a tool for quality appraisal of systematic reviews of prognostic factor studies
- Collapsing answer options by combining Yes with Probably Yes and No with Probably No improved agreement across all three levels of analysis. — Agreement testing of AMSTAR-PF, a tool for quality appraisal of systematic reviews of prognostic factor studies
- Earlier AMSTAR-PF questions covering review planning, literature searching, and study inclusion showed higher agreement than later questions addressing synthesis and interpretation, reflecting greater subjectivity in those domains. — Agreement testing of AMSTAR-PF, a tool for quality appraisal of systematic reviews of prognostic factor studies
- Cochrane reviews showed significantly higher intrapair agreement than non-Cochrane reviews, likely because stricter Cochrane reporting standards reduce the ambiguity appraisers must navigate. — Agreement testing of AMSTAR-PF, a tool for quality appraisal of systematic reviews of prognostic factor studies
- Agreement scores did not change significantly across the testing period, so improvements in completion time reflected gains in efficiency rather than rating accuracy. — Agreement testing of AMSTAR-PF, a tool for quality appraisal of systematic reviews of prognostic factor studies