检索结果-内蒙古大学图书馆

ROSE: Robust Selective Fine-tuning for Pre-trained Language Models

学校读者我要写书评

暂无评论

ROSE: Robust Selective Fine-tuning for Pre-trained Language ...

2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

作者： Jiang, Lan Zhou, Hao Lin, Yankai Li, Peng Zhou, Jie Jiang, Rui MOE Key Laboratory of Bioinformatics Center for Synthetic and Systems Biology Department of Automation BNRist Tsinghua University China Pattern Recognition Center WeChat AI Tencent Inc. China Gaoling School of Artificial Intelligence Renmin University of China Beijing China Beijing Key Laboratory of Big Data Management and Analysis Methods Beijing China Tsinghua University China

Even though the large-scale language models have achieved excellent performances, they suffer from various adversarial attacks. A large body of defense methods has been proposed. However, they are still limited due to redundant attack search spaces and the inability to defend against various types of attacks. In this work, we present a novel fine-tuning approach called RObust SEletive fine-tuning (ROSE) to address this issue. ROSE conducts selective updates when adapting pre-trained models to downstream tasks, filtering out invaluable and unrobust updates of parameters. Specifically, we propose two strategies: the first-order and second-order ROSE for selecting target robust parameters. The experimental results show that ROSE achieves significant improvements in adversarial robustness on various downstream NLP tasks, and the ensemble method even surpasses both variants above. Furthermore, ROSE can be easily incorporated into existing fine-tuning methods to improve their adversarial robustness further. The empirical analysis confirms that ROSE eliminates unrobust spurious updates during fine-tuning, leading to solutions corresponding to flatter and wider optima than the conventional method. Code is available at https://***/jiangllan/ROSE. © 2022 Association for Computational Linguistics.

关键词： Computational linguistics

Brain Fractal Dimension and Machine Learning can predict first-episode psychosis and risk for transition to psychosis

学校读者我要写书评

暂无评论

Computers in Biology and Medicine 2025年 193卷 110333-110333页

作者： Hu, Yaxin Frisman, Marina Andreou, Christina Avram, Mihai Riecher-Rössler, Anita Borgwardt, Stefan Barth, Erhardt Korda, Alexandra Institute of Neuro- and Bioinformatics University of Luebeck Ratzeburger Allee 160 Schleswig-Holstein Luebeck23562 Germany Pattern Recognition Company GmbH Maria-Goeppert-Straße 3 Schleswig-Holstein Luebeck23562 Germany Schleswig-Holstein University Medical Center Ratzeburger Allee 160 Schleswig-Holstein Luebeck23562 Germany Faculty of Medicine University of Basel Klingelbergstr. 61 CH Basel4056 Switzerland

Although there are notable structural abnormalities in the brain associated with psychotic diseases, it is still unclear how these abnormalities relate to clinical presentation. However, the fractal dimension (FD), which offers details on the complexity and irregularity of brain microstructures, may be a promising feature, as demonstrated by neuropsychiatric disorders such as Parkinson's and Alzheimer's. It may offer a possible biomarker for the detection and prognosis of psychosis when paired with machine learning. The purpose of this study is to investigate FD as a structural magnetic resonance imaging (sMRI) feature from individuals with a high clinical risk of psychosis who did not transit to psychosis (CHR_NT), clinical high risk who transit to psychosis (CHR_T), patients with first-episode psychosis (FEP) and healthy controls (HC). Using a machine learning approach that ultimately classifies sMRI images, the goals are (a) to evaluate FD as a potential biomarker and (b) to investigate its ability to predict a subsequent transition to psychosis from the high-risk clinical condition. We obtained sMRI images from 194 subjects, including 44 HCs, 77 FEPs, 16 CHR_Ts, and 57 CHR_NTs. We extracted the FD features and analyzed them using machine learning methods under five classification schemas (a) FEP vs. HC, (b) FEP vs. CHR_NT, (c) FEP vs. CHR_T, (d) CHR_NT vs. CHR_T, (d) CHR_NT vs. HC and (e) CHR_T vs. HC. In addition, the CHR_T group was used as external validation in (a), (b) and (d) comparisons to examine whether the progression of the disorder followed the FEP or CHR_NT patterns. The proposed algorithm resulted in a balanced accuracy greater than 0.77. This study has shown that FD can function as a predictive neuroimaging marker, providing fresh information on the microstructural alterations triggered throughout the course of psychosis. The effectiveness of FD in the detection of psychosis and transition to psychosis should be established by further research usi

关键词： Fractal dimension

iPINNs: Incremental learning for Physics-informed neural networks

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Dekhovich, Aleksandr Sluiter, Marcel H.F. Tax, David M.J. Bessa, Miguel A. Department of Materials Science and Engineering Delft University of Technology Mekelweg 2 Delft2628 CD Netherlands Pattern Recognition and Bioinformatics Laboratory Delft University of Technology Van Mourik Broekmanweg 6 Delft2628 XE Netherlands School of Engineering Brown University 184 Hope St. ProvidenceRI02912 United States

Physics-informed neural networks (PINNs) have recently become a powerful tool for solving partial differential equations (PDEs). However, finding a set of neural network parameters that lead to fulfilling a PDE can be challenging and non-unique due to the complexity of the loss landscape that needs to be traversed. Although a variety of multi-task learning and transfer learning approaches have been proposed to overcome these issues, there is no incremental training procedure for PINNs that can effectively mitigate such training challenges. We propose incremental PINNs (iPINNs) that can learn multiple tasks (equations) sequentially without additional parameters for new tasks and improve performance for every equation in the sequence. Our approach learns multiple PDEs starting from the simplest one by creating its own subnetwork for each PDE and allowing each subnetwork to overlap with previously learned subnetworks. We demonstrate that previous subnetworks are a good initialization for a new equation if PDEs share similarities. We also show that iPINNs achieve lower prediction error than regular PINNs for two different scenarios: (1) learning a family of equations (e.g., 1-D convection PDE);and (2) learning PDEs resulting from a combination of processes (e.g., 1-D reaction-diffusion PDE). The ability to learn all problems with a single network together with learning more complex PDEs with better generalization than regular PINNs will open new avenues in this field. © 2023, CC BY.

关键词： Complex networks

POS0750 WHO PARTICIPATES IN AN ONLINE SYMPTOM CHECKER? CHARACTERISTICS, ATTRITION AND REPRESENTATIVENESS IN THE RHEUMATIC? STUDY

学校读者我要写书评

暂无评论

Annals of the Rheumatic Diseases 2024年 83卷 1128-1129页

作者： F. Zegers L. Qin D. Selani G. Gomon T.D. Maarseveen H.K. Glas A. Van Tubergen Y. Goekoop-Ruiterman M.J.T. Reinders E.B. Van den Akker C. Baatenburg de Jong L. Klareskog B. Axnäs E. Steyerberg R. Bos S. Le Cessie R. Knevel Leiden University Medical Center Department of Biomedical Data Sciences Leiden Netherlands Leiden University Medical Center Department of Rheumatology Leiden Netherlands Delft University of Technology Department of Intelligent Systems Pattern Recognition and Bioinformatics Delft Netherlands Reumazorg Zuid West Neder；anmd Goes Netherlands Maastricht University Medical Center Department of Internal Medicine Division of Rheumatology Maastricht Netherlands Maastricht University Care and Public Health Research Institute (CAPHRI) Maastricht Netherlands Haga Hospital Department of Rheumatology Den Haag Netherlands Delft Technical University Delft Bioinformatics Lab Delft Netherlands Leiden University Medical Center Leiden Computational Biology Department of Biomedical Data Sciences Leiden Netherlands Delft University of Technology The Delft Bioinformatics Lab Pattern Recognition and Bioinformatics Delft Netherlands ReumaNederland Amsterdam Netherlands (Solna) Karolinska Institutet Division for Rheumatology Department of Medicine Stockholm Sweden Elsa Science Stockholm Sweden University Medical Center Rotterdam Department of Public Health Erasmus MC Rotterdam Netherlands Medical Center Leeuwarden Department of Rheumatology Leeuwarden Netherlands Nij Smellinghe Hospital Department of Rheumatology Drachten Netherlands Leiden University Medical Center Department of Clinical Epidemiology Leiden Netherlands Newcastle University Translational and Clinical Research Institute Rheumatology Newcastle upon Tyne United Kingdom

Background: Online symptom checkers (OSCs) could provide guidance in the diagnostic process. The number of OSCs are expanding yet often lack rigorous validation. Rheumatic? is an online questionnaire designed to provide a fast and reliable risk assessment of possible rheumatic diagnoses for individuals with musculoskeletal complaints (MSCs). We set out to validate this tool by following individuals with MSCs. A key question is representativity of the participants as attrition bias can complicate the construction of a precise prediction model. Objectives: This project aims to assess aspects of attrition and selection bias in Rheumatic? by comparing participants who did and did not respond to the first follow-up survey, by comparing participants versus patients from general practitioners (GP) and rheumatology clinics, and by exploring variations in characteristics across different recruitment policies. Methods: Rheumatic? includes a baseline questionnaire about current symptoms and past diagnoses. Participants were recruited online, at primary care or at rheumatology outpatient clinics. Four surveys followed after 0, 3, 6 and 12 months, about source of recruitment, medication, further diagnoses etc. We considered data of July 2021 to July 2023. We compared participants who solely answered the questionnaire to those who also answered the first follow up survey. Secondly, we compared characteristics of participants across different sources of recruitment (i.e. at primary or secondary care clinics or online (Instagram, Facebook, Google, the Dutch Arthritis Association)). For this, we calculated differences in percentages with confidence intervals and applied chi squared tests. We assessed representability of the cohort with the expected target population by comparing age, sex and BMI of the participants with information on new patients with MSC from the same period at primary care (GP), Rheumatology clinics in the South West of the Netherlands (RZWN), and the Rheumatolog

关键词：

Cooperative data-driven modeling

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Dekhovich, Aleksandr Turan, O. Taylan Yi, Jiaxiang Bessa, Miguel A. Department of Material Science and Engineering Delft University of Technology Mekelweg 2 Delft2628 CD Netherlands Pattern Recognition and Bioinformatics Laboratory Delft University of Technology Van Mourik Broekmanweg 6 Delft2628 XE Netherlands School of Engineering Brown University 184 Hope St. ProvidenceRI02912 United States

Data-driven modeling in mechanics is evolving rapidly based on recent machine learning advances, especially on artificial neural networks. As the field matures, new data and models created by different groups become available, opening possibilities for cooperative modeling. However, artificial neural networks suffer from catastrophic forgetting, i.e. they forget how to perform an old task when trained on a new one. This hinders cooperation because adapting an existing model for a new task affects the performance on a previous task trained by someone else. The authors developed a continual learning method1 that addresses this issue, applying it here for the first time to solid mechanics. In particular, the method is applied to recurrent neural networks to predict history-dependent plasticity behavior, although it can be used on any other architecture (feedforward, convolutional, etc.) and to predict other phenomena. This work intends to spawn future developments on continual learning that will foster cooperative strategies among the mechanics community to solve increasingly challenging problems. We show that the chosen continual learning strategy can sequentially learn several constitutive laws without forgetting them, using less data to achieve the same error as standard (non-cooperative) training of one law per model. © 2022, CC BY.

关键词： Recurrent neural networks

ROSE: Robust Selective Fine-tuning for Pre-trained Language Models

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Jiang, Lan Zhou, Hao Lin, Yankai Li, Peng Zhou, Jie Jiang, Rui Ministry of Education Key Laboratory of Bioinformatics Center for Synthetic and Systems Biology Department of Automation BNRist Tsinghua University China Pattern Recognition Center WeChat AI Tencent Inc. China Gaoling School of Artificial Intelligence Renmin University of China Beijing China Beijing Key Laboratory of Big Data Management and Analysis Methods Beijing China Tsinghua University China

Even though the large-scale language models have achieved excellent performances, they suffer from various adversarial attacks. A large body of defense methods has been proposed. However, they are still limited due to redundant attack search spaces and the inability to defend against various types of attacks. In this work, we present a novel fine-tuning approach called RObust SEletive fine-tuning (ROSE) to address this issue. ROSE conducts selective updates when adapting pretrained models to downstream tasks, filtering out invaluable and unrobust updates of parameters. Specifically, we propose two strategies: the first-order and second-order ROSE for selecting target robust parameters. The experimental results show that ROSE achieves significant improvements in adversarial robustness on various downstream NLP tasks, and the ensemble method even surpasses both variants above. Furthermore, ROSE can be easily incorporated into existing fine-tuning methods to improve their adversarial robustness further. The empirical analysis confirms that ROSE eliminates unrobust spurious updates during fine-tuning, leading to solutions corresponding to flatter and wider optima than the conventional method. Code is available at https://***/jiangllan/ROSE. © 2022, CC BY.

关键词： Computational linguistics

AB1448 DIFFERENTIATION OF IMMUNE MEDIATED VERSUS NON IMMUNE MEDIATED RHEUMATIC DISEASES BY ONLINE SYMPTOM CHECKER IN REAL-WORLD PATIENTS – MULTIPLE DIAGNOSES AND PARTICULARLY FIBROMYALGIA IS A STUMBLING BLOCK

学校读者我要写书评

暂无评论

Annals of the Rheumatic Diseases 2024年 83卷 2082-2083页

作者： L. Qin F. Zegers D. Selani E.B. Van den Akker R. Bos S. Le Cessie I. Gehring H.K. Glas M. Johannesson J. Knitza L. Mathsson-Alm M.J.T. Reinders E. Steyerberg A. Van Tubergen L.M. Verhoef B. Axnäs L. Klareskog R. Knevel Leiden University Medical Center Department of Rheumatology Leiden Netherlands Leiden University Medical Center Department of Biomedical Data Sciences Leiden Netherlands Delft University of Technology Department of Intelligent Systems Pattern Recognition and Bioinformatics Delft Netherlands Leiden University Medical Center Leiden Computational Biology Department of Biomedical Data Sciences Leiden Netherlands Delft University of Technology The Delft Bioinformatics Lab Pattern Recognition and Bioinformatics Delft Netherlands Medical Center Leeuwarden Department of Rheumatology Leeuwarden Netherlands Nij Smellinghe Hospital Department of Rheumatology Drachten Netherlands Leiden University Medical Center Department of Clinical Epidemiology Leiden Netherlands Thermo Fisher Scientific Freiburg Germany Reumazorg Zuid West Nederland Goes Netherlands Karolinska Institutet and Karolinska University Hospital Department of Medicine Solna Solna Sweden University Hospital of Giessen and Marburg Philipps-University Marburg Institute for Digital Medicine Marburg Germany Uppsala University Department of Immunology Genetics and Pathology Uppsala Sweden Thermo Fisher Scientific Uppsala Sweden University Medical Center Rotterdam Department of Public Health Rotterdam Netherlands Maastricht University Medical Center Department of Internal Medicine Division of Rheumatology Maastricht Netherlands Maastricht University Care and Public Health Research Institute (CAPHRI) Maastricht Netherlands Sint Maartenskliniek Research Department Ubbergen Netherlands Elsa Science Stockholm Sweden Karolinska Institutet and Karolinska University hospital Department of Medicine Solna Solna Sweden Newcastle University Translational and Clinical Research Institute Rheumatology Newcastle upon Tyne United Kingdom

Background: Rheumatic? is an online symptom checker developed collaboratively by designers, engineers, clinical experts and patients. It is designed to identify individuals at high risk of developing rheumatic diseases and patients with immune-mediated rheumatic diseases (IRD). After completion of the questionnaire, Rheumatic? gives a weighted score based on expert assessments of the clinical significance of the answers [1] . A previous retrospective study has demonstrated the potential to distinguish IRD from osteoarthritis(OA) and gout, where clinicians filled out the questionnaire based on the medical notes [2] . In order to use a questionnaire in real life, it should be tested on real life participants, as they may describe their symptoms differently than experts. In order to evaluate the performance of the current Rheumatic? score, to differentiate between IRDs and non-IRDs we conducted a prospective real-world study in the Netherlands and assessed the differences in Rheumatic? total scores between individuals diagnosed with IRD and non-IRD. Objectives: Examine the difference in scores of Rheumatic? between IRD (Rheumatoid arthritis, Psoriatic arthritis, Myositis, Sjogren’s syndrome, SLE, Spondyloarthritis and Polymyalgia rheumatica) and non-IRD(OA, Gout, Fibromyalgia and other) diagnoses. Methods: We analyzed data submitted by Dutch participants recruited via multiple channels between July 2021 and July 2022. After completing the online questionnaire Rheumatic? participants received four follow-up surveys at 1 week, 3, 6 and 12 months asking, among other things, for previous and novel diagnoses. The theoretical total score for the participant was determined using the current, expert developed, risk assessment model post collection of all data [1,2] . We used t-test and analysis of variance (ANOVA) to compared the observed differences in mean and median of total scores among these groups. Results: 8,727 participants were included, of which 6861(79%) were female

关键词：

Hybrid genetic-discretized algorithm to handle data uncertainty in diagnosing stenosis of coronary arteries

学校读者我要写书评

暂无评论

Hybrid genetic-discretized algorithm to handle data uncertai...

作者： Alizadehsani, Roohallah Roshanzamir, Mohamad Abdar, Moloud Beykikhoshk, Adham Khosravi, Abbas Nahavandi, Saeid Plawiak, Pawel Tan, Ru San Acharya, U Rajendra Deakin University GeelongVIC Australia Department of Engineering Fasa Branch Islamic Azad University Fars Fasa Iran Centre for Pattern Recognition and Data Analytics Deakin University GeelongVIC Australia Department of Information and Communications Technology Faculty of Computer Science and Telecommunications Cracow University of Technology Warszawska 24 st. F-3 31-155 Krakow Poland Institute of Theoretical and Applied Informatics Polish Academy of Sciences Baltycka 5 Gliwice44-100 Poland National Heart Centre Singapore Singapore Department of Electronics and Computer Engineering Ngee Ann Polytechnic Singapore Department of Biomedical Engineering School of Science and Technology Singapore University of Social Sciences Singapore Department of Bioinformatics and Medical Engineering Asia University Taiwan

Coronary artery disease (CAD) is the leading cause of morbidity and death worldwide. Invasive coronary angiography is the most accurate technique for diagnosing CAD, but is invasive and costly. Hence, analytical methods such as machine learning and data mining techniques are becoming increasingly more popular. Although physicians need to know which arteries are stenotic, most of the researchers focus only on CAD detection and few studies have investigated stenosis of the right coronary artery (RCA), left circumflex (LCX) artery and left anterior descending (LAD) artery separately. Meanwhile, most of the datasets in this field are noisy (data uncertainty). However, to the best of our knowledge, there is no study conducted to address this important problem. This study uses the extension of the Z-Alizadeh Sani dataset, containing 303 records with 54 features. A new feature selection algorithm is proposed in this work. Meanwhile, by discretization of data, we also handle the uncertainty in CAD prediction. To the best of our knowledge, this is the first study attempted to handle uncertainty in CAD prediction. Finally, the genetic algorithm (GA) is used to determine the hyper-parameters of the support vector machine (SVM) kernels. We have achieved high accuracy for the stenosis diagnosis of each main coronary artery. The results of this study can aid the clinicians to validate their manual stenosis diagnosis of RCA, LCX and LAD coronary arteries. © 2020 John Wiley & Sons, Ltd.

关键词： Feature Selection

Neural network relief: a pruning algorithm based on neural activity

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Dekhovich, Aleksandr Tax, David M.J. Sluiter, Marcel H.F. Bessa, Miguel A. Department of Materials Science and Engineering Delft University of Technology Mekelweg 2 Delft2628 CD Netherlands Pattern Recognition and Bioinformatics Laboratory Delft University of Technology Van Mourik Broekmanweg 6 Delft2628 XE Netherlands School of Engineering Brown University 184 Hope St. ProvidenceRI02912 United States

Current deep neural networks (DNNs) are overparameterized and use most of their neuronal connections during inference for each task. The human brain, however, developed specialized regions for different tasks and performs inference with a small fraction of its neuronal connections. We propose an iterative pruning strategy introducing a simple importance-score metric that deactivates unimportant connections, tackling overparameterization in DNNs and modulating the firing patterns. The aim is to find the smallest number of connections that is still capable of solving a given task with comparable accuracy, i.e. a simpler subnetwork. We achieve comparable performance for LeNet architectures on MNIST, and significantly higher parameter compression than state-of-the-art algorithms for VGG and ResNet architectures on CIFAR-10/100 and Tiny-ImageNet.∗ Our approach also performs well for the two different optimizers considered – Adam and SGD. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations, although it performs reasonably when compared to the state of the art. © 2021, CC BY.

关键词： Deep neural networks