检索结果-内蒙古大学图书馆

Memory3: Language Modeling with Explicit Memory

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yang, Hongkang Lin, Zehao Wang, Wenjin Wu, Hao Li, Zhiyu Tang, Bo Wei, Wenqiang Wang, Jinbo Tang, Zeyun Song, Shichao Xi, Chenyang Yu, Yu Chen, Kai Xiong, Feiyu Tang, Linpeng Weinan, E. Center for LLM Institute for Advanced Algorithms Research Shanghai China Moqi Inc China Center for Machine Learning Research Peking University China School of Mathematical Sciences Peking University AI for Science Institute China

The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining "abstract knowledge". As a preliminary proof of concept, we train from scratch a 2.4B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG. The model is named Memory3, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values). We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory *** Codes 68T50 Copyright © 2024, The Authors. All rights reserved.

关键词： Digital storage

CASSPR: Cross Attention Single Scan Place Recognition

学校读者我要写书评

暂无评论

CASSPR: Cross Attention Single Scan Place Recognition

International Conference on Computer Vision (ICCV)

作者： Yan Xia Mariia Gladkova Rui Wang Qianyun Li Uwe Stilla João F. Henriques Daniel Cremers Technical University of Munich Munich Center for Machine Learning (MCML) Visual Geometry Group University of Oxford Munich Data Science Institute Microsoft Zurich

Place recognition based on point clouds (LiDAR) is an important component for autonomous robots or self-driving vehicles. Current SOTA performance is achieved on accumulated LiDAR submaps using either point-based or voxel-based structures. While voxel-based approaches nicely integrate spatial context across multiple scales, they do not exhibit the local precision of point-based methods. As a result, existing methods struggle with fine-grained matching of subtle geometric features in sparse single-shot Li-DAR scans. To overcome these limitations, we propose CASSPR as a method to fuse point-based and voxel-based approaches using cross attention transformers. CASSPR leverages a sparse voxel branch for extracting and aggregating information at lower resolution and a point-wise branch for obtaining fine-grained local information. CASSPR uses queries from one branch to try to match structures in the other branch, ensuring that both extract self-contained descriptors of the point cloud (rather than one branch dominating), but using both to inform the out-put global descriptor of the point cloud. Extensive experiments show that CASSPR surpasses the state-of-the-art by a large margin on several datasets (Oxford RobotCar, TUM, USyd). For instance, it achieves AR@1 of 85.6% on the TUM dataset, surpassing the strongest prior model by ~15%. Our code is publicly available. 1

关键词：

Improved Approach for Mapping Anthropological Facial Features Based on A Convolutional Neural Network 2nd

学校读者我要写书评

暂无评论

Improved Approach for Mapping Anthropological Facial Feature...

2nd International Conference on Advances in Information and Communication Technology, ICTA 2023

作者： Huong, Nguyen Thu Long, Nguyen The Lien, Pham Thi Department of Informatics Institute of Cybersecurity and Digital Technologies MIREA – Russian Technological University Moscow119454 Russia Laboratory of Artificial Intelligence and Machine Learning Institute of Information Technology and Data Science Irkutsk National Research Technical University Irkutsk664074 Russia University of Information and Communication Technology Thai Nguyen University Thai Nguyen70000 Viet Nam

ISBN: (纸本)9783031495281

Our facial anthropological and actions are non-verbal communication tools that account for 93% of emotions in human communication, 55% of which are facial expressions and gestures. Facial emotions can be easily analyzed from facial images, and computers can interact with people the way people interact with each other. For this reason, automatic classification and analysis of facial emotions is currently an interesting and challenging problem that has great implications for society. In this work, the random forest method combined with a neural network was proposed to improve the classification of emotions by building feature maps. Training and testing models were built to perform the experiments, which achieved very good accuracy in the range of 98.5% to 99.7% for the two datasets. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

关键词： Convolution

DFingerNet: Noise-Adaptive Speech Enhancement for Hearing Aids

学校读者我要写书评

暂无评论

DFingerNet: Noise-Adaptive Speech Enhancement for Hearing Ai...

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Tsangko, Iosif Triantafyllopoulos, Andreas Müller, Michael Schröter, Hendrik Schuller, Björn W. EIHW - Embedded Intelligence for Health Care and Wellbeing University of Augsburg Germany Technical University of Munich Germany MCML - Munich Center for Machine Learning Munich Germany WS Audiology Research and Development Erlangen Germany GLAM - Group on Language Audio & Music Imperial College London United Kingdom MDSI - Munich Data Science Institute Munich Germany

ISBN: (纸本)9798350368741

The DeepFilterNet (DFN) architecture was recently proposed as a deep learning model suited for hearing aid devices. Despite its competitive performance on numerous benchmarks, it still follows a 'one-size-fits-all' approach, which aims to train a single, monolithic architecture that generalises across different noises and environments. However, its limited size and computation budget can hamper its generalisability. Recent work has shown that in-context adaptation can improve performance by conditioning the denoising process on additional information extracted from background recordings to mitigate this. These recordings can be offloaded outside the hearing aid, thus improving performance while adding minimal computational overhead. We introduce these principles to the DFN model, thus proposing the DFingerNet (DFiN) model, which shows superior performance on various benchmarks inspired by the DNS Challenge. © 2025 IEEE.

关键词： Context Adaptation Deep learning Hearing Aids Speech Denoising Speech Enhancement

Regularized Shannon sampling formulas related to the special affine Fourier transform

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Filbir, Frank Tasche, Manfred Veselovska, Anna Institute of Mathematics University of Rostock Germany Department of Mathematics Munich Data Science Institute Technical University of Munich Munich Center for Machine Learning Garching/Munich Germany

In this paper, we present new regularized Shannon sampling formulas related to the special affine Fourier transform (SAFT). These sampling formulas use localized sampling with special compactly supported window functions, namely B-spline, sinh-type, and continuous Kaiser–Bessel window functions. In contrast to the Shannon sampling series for SAFT, the regularized Shannon sampling formulas for SAFT possesses an exponential decay of the approximation error and are numerically robust in the presence of noise, if certain oversampling condition is fulfilled. Several numerical experiments illustrate the theoretical *** Codes 94A20, 42A38, 65T50 © 2023, CC BY.

关键词： Fourier transforms

HERALD: A NATURAL LANGUAGE ANNOTATED LEAN 4 dataSET

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Gao, Guoxiong Wang, Yutong Jiang, Jiedong Gao, Qi Qin, Zihan Xu, Tianyi Dong, Bin Peking University China National University of Singapore Singapore Center for Data Science Peking University China Beijing International Center for Mathematical Research The New Cornerstone Science Laboratory Peking University China Center for Machine Learning Research Peking University China

Verifiable formal languages like Lean have profoundly impacted mathematical reasoning, particularly through the use of large language models (LLMs) for automated reasoning. A significant challenge in training LLMs for these formal languages is the lack of parallel datasets that align natural language with formal language proofs. To address this challenge, this paper introduces a novel framework for translating the Mathlib4 corpus (a unified library of mathematics in formal language Lean 4) into natural language. Building upon this, we employ a dual augmentation strategy that combines tactic-based and informal-based approaches, leveraging the Lean-jixia system, a Lean 4 analyzer. We present the results of this pipeline on Mathlib4 as Herald (Hierarchy and Retrieval-based Translated Lean dataset). We also propose the Herald Translator, which is fine-tuned on Herald. Herald translator achieves a 93.2% accuracy (Pass@128) on formalizing statements in the miniF2F-test and a 22.5% accuracy on our internal graduate-level textbook dataset, outperforming InternLM2-Math-Plus-7B (74.0% and 7.5%) and TheoremLlama (50.1% and 4.0%). Furthermore, we propose a section-level translation framework for real-world applications. As a direct application of Herald translator, we have successfully translated a template section in the Stack project, marking a notable progress in the automatic formalization of graduate-level mathematical literature. Our model, along with the datasets, are open-sourced to the public.1 © 2024, CC BY.

关键词： Lean production

Predicting gastric cancer response to anti-HER2 therapy or anti-HER2 combined immunotherapy based on multimodal data

学校读者我要写书评

暂无评论

Signal Transduction and Targeted Therapy 2024年第9期9卷 4137-4148页

作者： Zifan Chen Yang Chen Yu Sun Lei Tang Li Zhang Yajie Hu Meng He Zhiwei Li Siyuan Cheng Jiajia Yuan Zhenghang Wang Yakun Wang Jie Zhao Jifang Gong Liying Zhao Baoshan Cao Guoxin Li Xiaotian Zhang Bin Dong Lin Shen Center for Data Science Peking UniversityBeijingChina Department of Gastrointestinal Oncology Key Laboratory of Carcinogenesis and Translational Research(Ministry of Education)Peking University Cancer Hospital and InstituteBeijingChina Department of Pathology Key Laboratory of Carcinogenesis and Translational Research(Ministry of Education)Peking University Cancer Hospital and InstituteBeijingChina Department of Radiology Key Laboratory of Carcinogenesis and Translational Research(Ministry of Education)Peking University Cancer Hospital and InstituteBeijingChina National Biomedical Imaging Center Peking UniversityBeijingChina Department of General Surgery Nanfang HospitalSouthern Medical UniversityGuangzhouChina Guangdong Provincial Key Laboratory of Precision Medicine for Gastrointestinal Tumor GuangzhouChina Department of Medical Oncology and Radiation Sickness Peking University Third HospitalBeijingChina National Engineering Laboratory for Big Data Analysis and Applications Peking UniversityBeijingChina Beijing International Center for Mathematical Research(BICMR) Peking UniversityBeijingChina Center for Machine Learning Research Peking UniversityBeijingChina

The sole use of single modality data often fails to capture the complex heterogeneity among patients,including the variability in resistance to anti-HER2 therapy and outcomes of combined treatment regimens,for the treatment of HER2-positive gastric cancer(GC).This modality deficit has not been fully considered in many ***,the application of artificial intelligence in predicting the treatment response,particularly in complex diseases such as GC,is still in its ***,this study aimed to use a comprehensive analytic approach to accurately predict treatment responses to anti-HER2 therapy or anti-HER2 combined immunotherapy in patients with HER2-positive *** collected multi-modal data,comprising radiology,pathology,and clinical information from a cohort of 429 patients:310 treated with anti-HER2 therapy and 119 treated with a combination of anti-HER2 and anti-PD-1/PD-L1 inhibitors *** introduced a deep learning model,called the Multi-Modal model(MuMo),that integrates these data to make precise treatment response *** achieved an area under the curve score of 0.821 for anti-HER2 therapy and 0.914 for combined ***,patients classified as low-risk by MuMo exhibited significantly prolonged progression-free survival and overall survival(log-rank test,P<0.05).These findings not only highlight the significance of multi-modal data analysis in enhancing treatment evaluation and personalized medicine for HER2-positive gastric cancer,but also the potential and clinical value of our model.

关键词： immunotherapy HER2 cancer

A causal intervention framework for synthesizing mobility data and evaluating predictive neural networks

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Hong, Ye Xin, Yanan Dirmeier, Simon Perez-Cruz, Fernando Raubal, Martin Institute of Cartography and Geoinformation ETH Zurich Switzerland Swiss Data Science Center ETH Zurich Switzerland EPFL Switzerland Institute for Machine Learning Department of Computer Science ETH Zurich Switzerland

Deep neural networks are increasingly utilized in mobility prediction tasks, yet their intricate internal workings pose challenges for interpretability, especially in comprehending how various aspects of mobility behavior affect predictions. This study introduces a causal intervention framework to assess the impact of mobility-related factors on neural networks designed for next location prediction – a task focusing on predicting the immediate next location of an individual. To achieve this, we employ individual mobility models to synthesize location visit sequences and control behavior dynamics by intervening in their data generation process. We evaluate the interventional location sequences using mobility metrics and input them into well-trained networks to analyze performance variations. The results demonstrate the effectiveness in producing location sequences with distinct mobility behaviors, thereby facilitating the simulation of diverse yet realistic spatial and temporal changes. These changes result in performance fluctuations in next location prediction networks, revealing impacts of critical mobility behavior factors, including sequential patterns in location transitions, proclivity for exploring new locations, and preferences in location choices at population and individual levels. The gained insights hold value for the real-world application of mobility prediction networks, and the framework is expected to promote the use of causal inference to enhance the interpretability and robustness of neural networks in mobility applications. © 2023, CC BY-NC-SA.

关键词： Prediction models

Exploring molecular pretraining model at scale 24

学校读者我要写书评

暂无评论

Exploring molecular pretraining model at scale

Proceedings of the 38th International Conference on Neural Information Processing Systems

作者： Xiaohong Ji Zhen Wang Zhifeng Gao Hang Zheng Linfeng Zhang Guolin Ke Weinan E DP Technology Beijing China DP Technology Beijing China and AI for Science Institute Beijing China AI for Science Institute Beijing China and School of Mathematical Sciences Peking University Beijing China and Center for Machine Learning Research Peking University Beijing China

ISBN: (纸本)9798331314385

In recent years, pretraining models have made significant advancements in the fields of natural language processing (NLP), computer vision (CV), and life sciences. The significant advancements in NLP and CV are predominantly driven by the expansion of model parameters and data size, a phenomenon now recognized as the scaling laws. However, research exploring scaling law in molecular pretraining models remains unexplored. In this work, we present Uni-Mol2, an innovative molecular pretraining model that leverages a two-track transformer to effectively integrate features at the atomic level, graph level, and geometry structure level. Along with this, we systematically investigate the scaling law within molecular pretraining models, characterizing the power-law correlations between validation loss and model size, dataset size, and computational resources. Consequently, we successfully scale Uni-Mol2 to 1.1 billion parameters through pretraining on 800 million conformations, making it the largest molecular pretraining model to date. Extensive experiments show consistent improvement in the downstream tasks as the model size grows. The Uni-Mol2 with 1.1B parameters also outperforms existing methods, achieving an average 27% improvement on the QM9 and 14% on COMPAS-1D dataset.

关键词：