检索结果-内蒙古大学图书馆

LaCo: Large Language Model Pruning via Layer Collapse

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yang, Yifei Cao, Zouying Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China

Large language models (LLMs) based on transformer are witnessing a notable trend of size expansion, which brings considerable costs to both model training and inference. However, existing methods such as model quantization, knowledge distillation, and model pruning are constrained by various issues, including hardware support limitations, the need for extensive training, and alterations to the model internal structure. In this paper, we propose a concise layer-wise structured pruner called Layer Collapse (LaCo), in which rear model layers collapse into a prior layer, enabling a rapid reduction in model size while preserving the model structure. Comprehensive experiments show that our method maintains an average task performance of over 80% at pruning ratios of 25-30%, significantly outperforming existing state-of-the-art structured pruning methods. We also conduct post-training experiments to confirm that the LaCo effectively inherits the parameters of the original model. Additionally, we perform ablation studies on various settings of LaCo. Finally, we discuss our motivation from the perspective of layer-wise similarity and evaluate the performance of the pruned LLMs across various pruning ratios Copyright © 2024, The Authors. All rights reserved.

关键词： Digital elevation model

Head-wise Shareable Attention for Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Cao, Zouying Yang, Yifei Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China

Large Language Models (LLMs) suffer from huge number of parameters, which restricts their deployment on edge devices. Weight sharing is one promising solution that encourages weight reuse, effectively reducing memory usage with less performance drop. However, current weight sharing techniques primarily focus on small-scale models like BERT and employ coarse-grained sharing rules, e.g., layer-wise. This becomes limiting given the prevalence of LLMs and sharing an entire layer or block obviously diminishes the flexibility of weight sharing. In this paper, we present a perspective on head-wise shareable attention for large language models. We further propose two memory-efficient methods that share parameters across attention heads, with a specific focus on LLMs. Both of them use the same dynamic strategy to select the shared weight matrices. The first method directly reuses the pre-trained weights without retraining, denoted as DirectShare. The second method first post-trains with constraint on weight matrix similarity and then shares, denoted as PostShare. Experimental results reveal our head-wise shared models still maintain satisfactory capabilities, demonstrating the feasibility of fine-grained weight sharing applied to LLMs1 Copyright © 2024, The Authors. All rights reserved.

关键词： Matrix algebra

Cross-system biological image quality enhancement based on the generative adversarial network as a foundation for establishing a multi-institute microscopy cooperative network

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Panek, Dominik Rząca, Carina Szczypior, Maksymilian Sorysz, Joanna Misztal, Krzysztof Baster, Zbigniew Rajfur, Zenon Doctoral School of Exact and Natural Sciences Jagiellonian University ul. Lojasiewicza 11 Kraków30-348 Poland Department of Molecular and Interfacial Biophysics Faculty of Physics Astronomy and Applied Computer Science Jagiellonian University ul. Lojasiewicza 11 Kraków30‑348 Poland Undergraduate Program in Biophysics Faculty of Physics Astronomy and Applied Computer Science Jagiellonian University ul. Lojasiewicza 11 Kraków30‑348 Poland Undergraduate program in Biomedical Engineering Faculty of Electrical Engineering Automatics Computer Science and Biomedical Engineering AGH University of Science and Technology Al. A. Mickiewicza 30 Building B-1 Kraków30-059 Poland Division of Computational Mathematics Faculty of Mathematics and Computer Science Jagiellonian University 6 Lojasiewicza St. Kraków Poland

High-quality fluorescence imaging of biological systems is limited by processes like photobleaching and phototoxicity, and also in many cases, by limited access to the latest generations of microscopes. Moreover, low temporal resolution can lead to a motion blur effect in living systems. Our work presents a deep learning (DL) generative-adversarial approach to the problem of obtaining high-quality (HQ) images based on their low-quality (LQ) equivalents. We propose a generative-adversarial network (GAN) for contrast transfer between two different separate microscopy systems: a confocal microscope (producing HQ images) and a wide-field fluorescence microscope (producing LQ images). Our model proves that such transfer is possible, allowing us to receive HQ-generated images characterized by low mean squared error (MSE) values, high structural similarity index (SSIM), and high peak signal-to-noise ratio (PSNR) values. For our best model in the case of comparing HQ-generated images and HQ-ground truth images, the median values of the metrics are 6·10-4, 0.9413, and 31.87, for MSE, SSIM, and PSNR, respectively. In contrast, in the case of comparison between LQ and HQ ground truth median values of the metrics are equal to 0.0071, 0.8304, and 21.48 for MSE, SSIM, and PSNR respectively. Therefore, we observe a significant increase ranging from 14% to 49% for SSIM and PSNR respectively. These results, together with other single-system cross-modality studies, provide proof of concept for further implementation of a cross-system biological image quality enhancement. © 2024, CC BY-SA.

关键词： Fluorescence imaging

Demonstration of a CAFQA-bootstrapped Variational Quantum Eigensolver on a trapped-ion quantum computer

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Qingfeng Zhukas, Liudmila Miao, Qiang Dalvi, Aniket S. Love, Peter J. Monroe, Christopher Chong, Frederic T. Ravi, Gokul Subramanian Duke Quantum Center Departments of Electrical and Computer Engineering and Physics Duke University Durham United States Chemical Physics Program Institute for Physical Science and Technology University of Maryland College ParkMD United States Duke Quantum Center Duke University Durham United States Department of Electrical and Computer Engineering Duke University Durham United States Department of Physics and Astronomy Tufts University Medford United States Department of Computer Science University of Chicago Chicago United States Department of Computer Science and Engineering University of Michigan Ann Arbor United States

To enhance the variational quantum eigensolver (VQE), the CAFQA method can utilize classical computational capabilities to identify a better initial state than the Hartree-Fock method. Previous research has demonstrated that the initial state provided by CAFQA recovers more correlation energy than that of the Hartree-Fock method and results in faster convergence. In the present study, we advance the investigation of CAFQA by demonstrating its advantages on a high-fidelity trapped-ion quantum computer located at the Duke Quantum Center—this is the first experimental demonstration of CAFQA-bootstrapped VQE on a TI device and on any academic quantum device. In our VQE experiment, we use LiH and BeH2 as test cases to show that CAFQA achieves faster convergence and obtains lower energy values within the specified computational budget limits. To ensure the seamless execution of VQE on this academic device, we develop a novel hardware-software interface framework that supports independent software environments for both the circuit and hardware end. This mechanism facilitates the automation of VQE-type job executions as well as mitigates the impact of random hardware interruptions. This framework is versatile and can be applied to a variety of academic quantum devices beyond the trapped-ion quantum computer platform, with support for integration with customized packages. © 2024, CC BY.

关键词： Quantum computers

Automating the Practice of science – Opportunities, Challenges, and Implications

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Musslick, Sebastian Bartlett, Laura K. Chandramouli, Suyog H. Dubova, Marina Gobet, Fernand Griffiths, Thomas L. Hullman, Jessica King, Ross D. Nathan Kutz, J. Lucas, Christopher G. Mahesh, Suhas Pestilli, Franco Sloman, Sabina J. Holmes, William R. Institute of Cognitive Science Osnabrück University Osnabrück49090 Germany Department of Cognitive Linguistic & Psychological Sciences Brown University ProvidenceRI02912 United States Centre for Philosophy of Natural and Social Science London School of Economics Lakatos Building Houghton Street LondonWC2A 2AE United Kingdom AaltoFI-00076 Finland Department of Computing Science University of Alberta 8900 114 St NW EdmontonABT6G 2S4 Canada Cognitive Science Program Indiana University 1101 E 10th St BloomingtonIN47405 United States School of Psychology University of Roehampton LondonSW15 4JD United Kingdom Departments of Psychology and Computer Science Princeton University PrincetonNJ United States Department of Computer Science Northwestern University IL United States Department of Chemical Engineering and Biotechnology University of Cambridge CambridgeCB3 0AS United Kingdom Department of Computer Science and Engineering Chalmers University of Technology Gothenburg412 96 Sweden Department of Applied Mathematics and Electrical and Computer Engineering University of Washington Seattle98195 United States School of Informatics University of Edinburgh 10 Crichton St. EH8 9AB United Kingdom Department of Materials Science and Engineering University of Toronto Canada Department of Psychology Department of Neuroscience The University of Texas AustinTX United States Department of Computer Science University of Manchester M13 9PL United Kingdom

Automation transformed various aspects of our human civilization, revolutionizing industries and streamlining processes. In the domain of scientific inquiry, automated approaches emerged as powerful tools, holding promise for accelerating discovery, enhancing reproducibility, and overcoming the traditional impediments to scientific progress. This article evaluates the scope of automation within scientific practice and assesses recent approaches. Furthermore, it discusses different perspectives to the following questions: Where do the greatest opportunities lie for automation in scientific practice?;What are the current bottlenecks of automating scientific practice?;and What are significant ethical and practical consequences of automating scientific practice? By discussing the motivations behind automated science, analyzing the hurdles encountered, and examining its implications, this article invites researchers, policymakers, and stakeholders to navigate the rapidly evolving frontier of automated scientific practice. © 2024, CC BY.

关键词：

Drone Application Model for Image Acquisition of Plantation Areas and Oil Palm Trees Counting 7

学校读者我要写书评

暂无评论

Drone Application Model for Image Acquisition of Plantation ...

7th International Conference on Information Management and Technology, ICIMTech 2022

作者： Sastrohartono, Hermantoro Suryotomo, Andiko Putro Saifullah, Shoffan Suparyanto, Teddy Perbangsa, Anzaludin Samsinga Pardamean, Bens Instiper Yogyakarta Faculty of Agricultural Technology Department of Agricultural Engineering Yogyakarta55281 Indonesia Universitas Pembangunan Nasional Veteran Yogyakarta Department of Informatics Yogyakarta55283 Indonesia Bioinformatics and Data Science Research Center Bina Nusantara University Jakarta11480 Indonesia School of Information Systems Bina Nusantara University Information Systems Department Jakarta11480 Indonesia BINUS Graduate Program - Master of Computer Science Bina Nusantara University Computer Science Department Jakarta11480 Indonesia

ISBN: (数字)9781665450904

ISBN: (纸本)9781665450904

The area of oil palm plantations in Indonesia increased by 7% from 14 million ha in 2017 to 15 million ha in 2021. The vast land requires the support of effective and efficient management techniques to maintain sustainable productivity. The high-performance computing technologies, Internet of Things (IoT), Big Data, Artificial Intelligence, spatial modeling, and drones are the answers to these needs. This study aims to design an application that can obtain plantation image data and analyze the calculation of the number of oil palm trees. The image of oil palm plantations is obtained from processing photo data from drones through a mosaic and composite process. This study also employed Scrum and UML as a system model development method. The image of the oil palm plantation area is used to build a tree counting model using the Viola-Jones algorithm. The oil palm tree count information generated by this system can then be used by management for fertilizing, harvesting, and monitoring the condition of oil palm trees. © 2022 IEEE.

关键词： Drones

Cell2Sentence: Teaching Large Language Models the Language of Biology 41

学校读者我要写书评

暂无评论

Cell2Sentence: Teaching Large Language Models the Language o...

41st International Conference on Machine Learning, ICML 2024

作者： Levine, Daniel Rizvi, Syed Asad Lévy, Sacha Pallikkavaliyaveetil, Nazreen Zhang, David Chen, Xingyu Ghadermarzi, Sina Wu, Ruiming Zheng, Zihe Vrkic, Ivan Zhong, Anna Raskin, Daphne Han, Insu de Oliveira Fonseca, Antonio Henrique Caro, Josue Ortega Karbasi, Amin Dhodapkar, Rahul M. van Dijk, David Department of Computer Science Yale University New HavenCT United States School of Engineering Applied Science University of Pennsylvania PhiladelphiaPA United States School of Computer and Communication Sciences Swiss Federal Institute of Technology Lausanne Lausanne Switzerland Department of Neuroscience Yale School of Medicine New HavenCT United States Wu Tsai Institute Yale University New HavenCT United States Google United States Yale Institute for Foundations of Data Science New HavenCT United States Yale School of Engineering and Applied Science New HavenCT United States Roski Eye Institute University of Southern California Los AngelesCA United States Yale School of Medicine New HavenCT United States Cardiovascular Research Center Yale School of Medicine New HavenCT United States Interdepartmental Program in Computational Biology & Bioinformatics Yale University New HavenCT United States

We introduce Cell2Sentence (C2S), a novel method to directly adapt large language models to a biological context, specifically single-cell transcriptomics. By transforming gene expression data into"cell sentences," C2S bridges the gap between natural language processing and biology. We demonstrate cell sentences enable the finetuning of language models for diverse tasks in biology, including cell generation, complex cell-type annotation, and direct data-driven text generation. Our experiments reveal that GPT-2, when fine-tuned with C2S, can generate biologically valid cells based on cell type inputs, and accurately predict cell types from cell sentences. This illustrates that language models, through C2S finetuning, can acquire a significant understanding of single-cell biology while maintaining robust text generation capabilities. C2S offers a flexible, accessible framework to integrate natural language processing with transcriptomics, utilizing existing models and libraries for a wide range of biological applications. Copyright 2024 by the author(s)

关键词： Gene expression

Improving Land Cover Segmentation Using Multispectral Dataset

学校读者我要写书评

暂无评论

Improving Land Cover Segmentation Using Multispectral Datase...

International Conference on Informatics and Computing (ICIC)

作者： Herlawati Herlawati Rahmadya Trias Handayanto Yaya Heryadi Edi Abdurachman Haryono Soeparno Edy Irwansyah Eka Miranda Informatics Universitas Bhayangkara Jakarta Raya Jakarta Indonesia Computer Engineering Universitas Islam 45 Bekasi Indonesia Computer Science Department BINUS Graduate Program – Doctor of Computer Science Bina Nusantara University Jakarta Indonesia Trisakti Institute of Transportation and Logistic Jakarta Indonesia Computer Science Department School of Computer Science Bina Nusantara University Jakarta Indonesia Information Systems Department School of Information Systems Bina Nusantara University Jakarta Indonesia

computer vision has been used in many areas such as medical, transportation, military, geography, etc. The fast development of sensor devices inside camera and satellite provides not only red-greed-blue (RGB) images but also multispectral dataset with some channels including RGB, infrared, short-wave, and thermal wave. Most of the dataset is panchromatic (black and white) and RGB, for example Google Map and other satellite-based map applications. This study examines the effects of multispectral dataset for semantic segmentation of land cover. The comparison between RGB with band 2 to band 7 of Landsat 8 Satellite shows an improvement of accuracy from 90.283 to 94.473 for U-Net and from 91.76 to 95.183 for DeepLabV3+. In addition, this research also compares two well-known semantic segmentation methods, namely U-Net and DeepLabV3+, that shown that DeepLabV3+ outperformed U-Net regarding to speed and accuracy. Testing was conducted in the Karawang Regency area, West Java, Indonesia.

关键词：

Aggregating Structural Features for Computational Analysis of Mutations in BRCT Domains of BRCA1 Protein

学校读者我要写书评

暂无评论

Aggregating Structural Features for Computational Analysis o...

International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS)

作者： Alam Ahmad Hidayat Rudi Nirwantono Mahmud Isnan Joko Pebrianto Trinugroho Bens Pardamean Mathematics Department School of Computer Science Bioinformatics and Data Science Research Center Bina Nusantara University Jakarta Indonesia Biotechnology Department Faculty of Engineering Bioinformatics and Data Science Research Center Bina Nusantara University Jakarta Indonesia Bioinformatics and Data Science Research Center Bina Nusantara University Jakarta Indonesia Computer Science Department Bioinformatics and Data Science Research Center BINUS Graduate Program Jakarta Indonesia

Understanding the mechanistic interpretability of mutation effects in a protein can help predict the clinical implications of the genetic variants. Hence, computational variant effect predictions that involve protein structural features of the protein mutations might be suitable in this case. In this work, we focus on BRCT domains of BRCA1 gene that is widely studied in breast cancer studies. We retrieved 88 selected missense variants found in BRCT domains annotated in both ClinVar and gnomAD databases. To computationally characterize the pathogenic property of the mutations we used two types of features extracted from protein structures: a change in free Gibbs energy and a set of features derived from molecular dynamics simulations of each mutant. Using a dimensional reduction and Gaussian mixture model (GMM)-based clustering we demonstrate that the variants are segregated into two regions that may correspond to their pathogenic status. This method can be a potential computational pipeline for providing the preliminary mechanistic interpretation of mutation effects in terms of their thermodynamic and structural features.

关键词：