The Universe of Utterances According to BERT

Dmitry Nikolaev, Sebastian Padó

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

It has been argued that BERT “rediscovers the traditional NLP pipeline”, with lower layers extracting morphosyntactic features and higher layers creating holistic sentence-level representations. In this paper, we critically examine this assumption through a principle-component-guided analysis, extracing sets of inputs that correspond to specific activation patterns in BERT sentence representations. We find that even in higher layers, the model mostly picks up on a variegated bunch of low-level features, many related to sentence complexity, that presumably arise from its specific pre-training objectives.
Original languageEnglish
Title of host publicationProceedings of the 15th International Conference on Computational Semantics
Place of PublicationNancy, France
PublisherAssociation for Computational Linguistics
Pages99-105
Publication statusPublished - Jun 2023
Externally publishedYes
EventThe 15th International Conference on Computational Semantics - Nancy, France
Duration: 21 Jun 202323 Jun 2023

Conference

ConferenceThe 15th International Conference on Computational Semantics
Abbreviated titleIWCS
Country/TerritoryFrance
CityNancy
Period21/06/2323/06/23

Fingerprint

Dive into the research topics of 'The Universe of Utterances According to BERT'. Together they form a unique fingerprint.

Cite this