What is SemEval evaluating?: A Systematic Analysis of Evaluation Campaigns in NLP

Oskar Wysocki; Malina Florea; Andre Freitas

doi:10.48550/arXiv.2005.14299

What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

Oskar Wysocki, Malina Florea, Andre Freitas

Information Management

Research output: Contribution to journal › Article

81 Downloads (Pure)

Abstract

SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.

Original language	English
Journal	ArXiv
DOIs	https://doi.org/10.48550/arXiv.2005.14299
Publication status	Published - 28 May 2020

Keywords

cs.CL
cs.AI

Access to Document

10.48550/arXiv.2005.14299

2005.14299v1

Cite this

@article{7683cbca3641438c89d3652e09d041bd,

title = "What is SemEval evaluating?: A Systematic Analysis of Evaluation Campaigns in NLP",

abstract = " SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval. ",

keywords = "cs.CL, cs.AI",

author = "Oskar Wysocki and Malina Florea and Andre Freitas",

note = "12 pages, 6 figures",

year = "2020",

month = may,

day = "28",

doi = "10.48550/arXiv.2005.14299",

language = "English",

journal = "ArXiv",

issn = "2331-8422",

publisher = "Cornell University",

}

TY - JOUR

T1 - What is SemEval evaluating?

T2 - A Systematic Analysis of Evaluation Campaigns in NLP

AU - Wysocki, Oskar

AU - Florea, Malina

AU - Freitas, Andre

N1 - 12 pages, 6 figures

PY - 2020/5/28

Y1 - 2020/5/28

N2 - SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.

AB - SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.

KW - cs.CL

KW - cs.AI

U2 - 10.48550/arXiv.2005.14299

DO - 10.48550/arXiv.2005.14299

M3 - Article

SN - 2331-8422

JO - ArXiv

JF - ArXiv

ER -

What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

Abstract

Keywords

Access to Document

Fingerprint

Cite this