Is the Understanding of Explicit Discourse Relations Required in Machine Reading Comprehension?

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

An in-depth analysis of the level of language understanding required by existing Machine Reading Comprehension (MRC) benchmarks can provide insight into the reading capabilities of machines. In this paper, we propose an ablation-based methodology to assess the extent to which MRC datasets evaluate the understanding of explicit discourse relations. We define seven MRC skills which require the understanding of different discourse relations. We then introduce ablation methods that verify whether these skills are required to succeed on a dataset. By observing the drop in performance of neural MRC models evaluated on the original and the modified dataset, we can measure to what degree the dataset requires these skills, in order to be understood correctly. Experiments on three large-scale datasets with the BERT-base and ALBERT-xxlarge model show that the relative changes for all skills are small (less than 6%). These results imply that most of the answered questions in the examined datasets do not require understanding the discourse structure of the text. To specifically probe for natural language understanding, there is a need to design more challenging benchmarks that can correctly evaluate the intended skills.
Original languageEnglish
Title of host publicationProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics
Subtitle of host publicationMain Volume
PublisherAssociation for Computational Linguistics
Pages3565-3579
DOIs
Publication statusPublished - 1 Apr 2021
EventProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume - Online
Duration: 1 Apr 20211 Apr 2021

Conference

ConferenceProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Period1/04/211/04/21

Fingerprint

Dive into the research topics of 'Is the Understanding of Explicit Discourse Relations Required in Machine Reading Comprehension?'. Together they form a unique fingerprint.

Cite this