Evaluating discourse and dialogue coding schemes

Richard Craggs; Mary Mcgee Wood

doi:10.1162/089120105774321109

Evaluating discourse and dialogue coding schemes

Richard Craggs, Mary Mcgee Wood

Research output: Contribution to journal › Article › peer-review

Abstract

Agreement statistics play an important role in the evaluation of coding schemes for discourse and dialogue. Unfortunately there is a lack of understanding regarding appropriate agreement measures and how their results should be interpreted. In this article we describe the role of agreement measures and argue that only chance-corrected measures that assume a common distribution of labels for all coders are suitable for measuring agreement in reliability studies. We then provide recommendations for how reliability should be inferred from the results of agreement statistics. © 2005 Association for Computational Linguistics.

Original language	English
Pages (from-to)	289-295
Number of pages	6
Journal	Computational Linguistics
Volume	31
Issue number	3
DOIs	https://doi.org/10.1162/089120105774321109
Publication status	Published - 1 Sept 2005

Access to Document

10.1162/089120105774321109

http://www.mitpressjournals.org/doi/pdf/10.1162/089120105774321109

Cite this

@article{f99dfa3654714570a188b6401f105c6a,

title = "Evaluating discourse and dialogue coding schemes",

abstract = "Agreement statistics play an important role in the evaluation of coding schemes for discourse and dialogue. Unfortunately there is a lack of understanding regarding appropriate agreement measures and how their results should be interpreted. In this article we describe the role of agreement measures and argue that only chance-corrected measures that assume a common distribution of labels for all coders are suitable for measuring agreement in reliability studies. We then provide recommendations for how reliability should be inferred from the results of agreement statistics. {\textcopyright} 2005 Association for Computational Linguistics.",

author = "Richard Craggs and Wood, {Mary Mcgee}",

year = "2005",

month = sep,

day = "1",

doi = "10.1162/089120105774321109",

language = "English",

volume = "31",

pages = "289--295",

journal = "Computational Linguistics",

issn = "0891-2017",

publisher = "MIT Press Journals",

number = "3",

}

TY - JOUR

T1 - Evaluating discourse and dialogue coding schemes

AU - Craggs, Richard

AU - Wood, Mary Mcgee

PY - 2005/9/1

Y1 - 2005/9/1

N2 - Agreement statistics play an important role in the evaluation of coding schemes for discourse and dialogue. Unfortunately there is a lack of understanding regarding appropriate agreement measures and how their results should be interpreted. In this article we describe the role of agreement measures and argue that only chance-corrected measures that assume a common distribution of labels for all coders are suitable for measuring agreement in reliability studies. We then provide recommendations for how reliability should be inferred from the results of agreement statistics. © 2005 Association for Computational Linguistics.

AB - Agreement statistics play an important role in the evaluation of coding schemes for discourse and dialogue. Unfortunately there is a lack of understanding regarding appropriate agreement measures and how their results should be interpreted. In this article we describe the role of agreement measures and argue that only chance-corrected measures that assume a common distribution of labels for all coders are suitable for measuring agreement in reliability studies. We then provide recommendations for how reliability should be inferred from the results of agreement statistics. © 2005 Association for Computational Linguistics.

U2 - 10.1162/089120105774321109

DO - 10.1162/089120105774321109

M3 - Article

SN - 0891-2017

VL - 31

SP - 289

EP - 295

JO - Computational Linguistics

JF - Computational Linguistics

IS - 3

ER -

Evaluating discourse and dialogue coding schemes

Abstract

Access to Document

Fingerprint

Cite this