How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modeling

作     者：Cahyawijaya, Samuel Wilie, Bryan Lovenia, Holy Zhong, MingQian Zhong, Huan Ip, Nancy Y. Fung, Pascale 

作者机构： Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology Hong Kong Hong Kong Division of Life Science State Key Laboratory of Molecular Neuroscience Molecular Neuroscience Center The Hong Kong University of Science and Technology Clear Water Bay Hong Kong Hong Kong Hong Kong Center for Neurodegenerative Diseases Hong Kong Science Park Hong Kong Hong Kong 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2022年

核心收录：

主　　题：Modeling languages 

摘      要：Large pre-trained language models (LMs) have been widely adopted in biomedical and clinical domains, introducing many powerful LMs such as bio-lm and BioELECTRA. However, the applicability of these methods to real clinical use cases is hindered, due to the limitation of pre-trained LMs in processing long textual data with thousands of words, which is a common length for a clinical note. In this work, we explore long-range adaptation from such LMs with Longformer, allowing the LMs to capture longer clinical notes context. We conduct experiments on three n2c2 challenges datasets and a longitudinal clinical dataset from Hong Kong Hospital Authority electronic health record (EHR) system to show the effectiveness and generalizability of this concept, achieving 10% F1-score improvement. Based on our experiments, we conclude that capturing a longer clinical note interval is beneficial to the model performance, but there are different cut-off intervals to achieve the optimal performance for different target variables. © 2022, CC BY-NC-SA.