The Universe of Utterances According to BERT

Dmitry Nikolaev; Sebastian Padó

The Universe of Utterances According to BERT

Dmitry Nikolaev
, Sebastian Padó

Linguistics and English Language (L5)

University of Stuttgart

Research output: Chapter in Book/Conference proceeding › Conference contribution › peer-review

Abstract

It has been argued that BERT “rediscovers the traditional NLP pipeline”, with lower layers extracting morphosyntactic features and higher layers creating holistic sentence-level representations. In this paper, we critically examine this assumption through a principle-component-guided analysis, extracing sets of inputs that correspond to specific activation patterns in BERT sentence representations. We find that even in higher layers, the model mostly picks up on a variegated bunch of low-level features, many related to sentence complexity, that presumably arise from its specific pre-training objectives.

Original language	English
Title of host publication	Proceedings of the 15th International Conference on Computational Semantics
Place of Publication	Nancy, France
Publisher	Association for Computational Linguistics
Pages	99-105
Publication status	Published - Jun 2023
Event	The 15th International Conference on Computational Semantics - Nancy, France Duration: 21 Jun 2023 → 23 Jun 2023

Conference

Conference	The 15th International Conference on Computational Semantics
Abbreviated title	IWCS
Country/Territory	France
City	Nancy
Period	21/06/23 → 23/06/23

Access to Document

https://aclanthology.org/2023.iwcs-1.12

Cite this

@inproceedings{9aaec0da2cf840cdb9d91b4bf7ca4e65,

title = "The Universe of Utterances According to BERT",

abstract = "It has been argued that BERT “rediscovers the traditional NLP pipeline”, with lower layers extracting morphosyntactic features and higher layers creating holistic sentence-level representations. In this paper, we critically examine this assumption through a principle-component-guided analysis, extracing sets of inputs that correspond to specific activation patterns in BERT sentence representations. We find that even in higher layers, the model mostly picks up on a variegated bunch of low-level features, many related to sentence complexity, that presumably arise from its specific pre-training objectives.",

author = "Dmitry Nikolaev and Sebastian Pad{\'o}",

year = "2023",

month = jun,

language = "English",

pages = "99--105",

booktitle = "Proceedings of the 15th International Conference on Computational Semantics",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "The 15th International Conference on Computational Semantics, IWCS ; Conference date: 21-06-2023 Through 23-06-2023",

}

TY - GEN

T1 - The Universe of Utterances According to BERT

AU - Nikolaev, Dmitry

AU - Padó, Sebastian

PY - 2023/6

Y1 - 2023/6

N2 - It has been argued that BERT “rediscovers the traditional NLP pipeline”, with lower layers extracting morphosyntactic features and higher layers creating holistic sentence-level representations. In this paper, we critically examine this assumption through a principle-component-guided analysis, extracing sets of inputs that correspond to specific activation patterns in BERT sentence representations. We find that even in higher layers, the model mostly picks up on a variegated bunch of low-level features, many related to sentence complexity, that presumably arise from its specific pre-training objectives.

AB - It has been argued that BERT “rediscovers the traditional NLP pipeline”, with lower layers extracting morphosyntactic features and higher layers creating holistic sentence-level representations. In this paper, we critically examine this assumption through a principle-component-guided analysis, extracing sets of inputs that correspond to specific activation patterns in BERT sentence representations. We find that even in higher layers, the model mostly picks up on a variegated bunch of low-level features, many related to sentence complexity, that presumably arise from its specific pre-training objectives.

M3 - Conference contribution

SP - 99

EP - 105

BT - Proceedings of the 15th International Conference on Computational Semantics

PB - Association for Computational Linguistics

CY - Nancy, France

T2 - The 15th International Conference on Computational Semantics

Y2 - 21 June 2023 through 23 June 2023

ER -

The Universe of Utterances According to BERT

Abstract

Conference

Access to Document

Fingerprint

Cite this