Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning

Lifeng Han, Gleb Erofeev, Irina Sorokina, Serge Gladkoff, Goran Nenadic

Research output: Contribution to conferencePaperpeer-review

Abstract

Massively multilingual pre-trained language models (MMPLMs) are developed in recent years demonstrating superpowers and the pre-knowledge they acquire for downstream tasks. This work investigates whether MMPLMs can be applied to clinical domain machine translation (MT) towards entirely unseen languages via transfer learning. We carry out an experimental investigation using Meta-AI's MMPLMs "wmt21-dense24-wide-en-X and X-en (WMT21fb)"which were pre-trained on 7 language pairs and 14 translation directions including English to Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese, and the opposite direction. We fine-tune these MMPLMs towards English-Spanish language pair which did not exist at all in their original pre-trained corpora both implicitly and explicitly. We prepare carefully aligned clinical domain data for this fine-tuning, which is different from their original mixed domain knowledge. Our experimental result shows that the fine-tuning is very successful using just 250k well-aligned in-domain ENES segments for three sub-task translation testings: clinical cases, clinical terms, and ontology concepts. It achieves very close evaluation scores to another MMPLM NLLB from Meta-AI, which included Spanish as a high-resource setting in the pre-training. To the best of our knowledge, this is the first work on using MMPLMs towards clinical domain transferlearning NMT successfully for totally unseen languages during pre-training.

Original languageEnglish
Pages31-40
Number of pages10
DOIs
Publication statusPublished - 2023
EventProceedings of the 5th Clinical Natural Language Processing Workshop - Toronto, Canada
Duration: 1 Jul 20231 Jul 2023

Conference

ConferenceProceedings of the 5th Clinical Natural Language Processing Workshop
Period1/07/231/07/23

Fingerprint

Dive into the research topics of 'Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning'. Together they form a unique fingerprint.

Cite this