检索结果-内蒙古大学图书馆

arXiv 2025年

作者： Chen, Yiming Zhang, Yuan Liu, Yin Yuan, Kun Wen, Zaiwen Beijing International Center for Mathematical Research Peking University Beijing China Center for Data Science Peking University Beijing China Center for Machine Learning Research Peking University Beijing China

The memory challenges associated with training Large Language Models (LLMs) have become a critical concern, particularly when using the Adam optimizer. To address this issue, numerous memory-efficient techniques have been proposed, with GaLore standing out as a notable example designed to reduce the memory footprint of optimizer states. However, these approaches do not alleviate the memory burden imposed by activations, rendering them unsuitable for scenarios involving long context sequences or large mini-batches. Moreover, their convergence properties are still not well-understood in the literature. In this work, we introduce a Randomized Subspace Optimization framework for pre-training and fine-tuning LLMs. Our approach decomposes the high-dimensional training problem into a series of lower-dimensional subproblems. At each iteration, a random subspace is selected, and the parameters within that subspace are optimized. This structured reduction in dimensionality allows our method to simultaneously reduce memory usage for both activations and optimizer states. We establish comprehensive convergence guarantees and derive rates for various scenarios, accommodating different optimization strategies to solve the subproblems. Extensive experiments validate the superior memory and communication efficiency of our method, achieving performance comparable to GaLore and Adam. Copyright © 2025, The Authors. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

On the Effectiveness of Heterogeneous Ensemble Methods for Re-identification

arXiv

引用

arXiv 2024年

作者： Klüttermann, Simon Rutinowski, Jérôme Nguyen, Anh Grimme, Britta Roidl, Moritz Müller, Emmanuel TU Dortmund University Dortmund Germany Lamarr Institute for Machine Learning and Artificial Intelligence Germany Research Center Trustworthy Data Science and Security Germany

In this contribution, we introduce a novel ensemble method for the re-identification of industrial entities, using images of chipwood pallets and galvanized metal plates as dataset examples. Our algorithms replace commonly used, complex siamese neural networks with an ensemble of simplified, rudimentary models, providing wider applicability, especially in hardware-restricted scenarios. Each ensemble sub-model uses different types of extracted features of the given data as its input, allowing for the creation of effective ensembles in a fraction of the training duration needed for more complex state-of-the-art models. We reach state-of-the-art performance at our task, with a Rank-1 accuracy of over 77% and a Rank-10 accuracy of over 99%, and introduce five distinct feature extraction approaches, and study their combination using different ensemble methods. © 2024, CC BY.

关键词： Complex networks

来源：评论

学校读者我要写书评

暂无评论

TVINESYNTH: A TRUNCATED C-VINE COPULA GENERATOR OF SYNTHETIC TABULAR data TO BALANCE PRIVACY AND UTILITY

arXiv

引用

arXiv 2025年

作者： Griesbauer, Elisabeth Czado, Claudia Frigessi, Arnoldo Haff, Ingrid Hobæk Norwegian Computing Center University of Oslo Integreat - Norwegian Centre for Knowledge-driven Machine Learning Norway Technical University of Munich Munich Data Science Institute Germany University of Oslo Integreat - Norwegian Centre for Knowledge-driven Machine Learning Norway

We propose TVineSynth, a vine copula based synthetic tabular data generator, which is designed to balance privacy and utility, using the vine tree structure and its truncation to do the trade-off. Contrary to synthetic data generators that achieve DP by globally adding noise, TVineSynth performs a controlled approximation of the estimated data generating distribution, so that it does not suffer from poor utility of the resulting synthetic data for downstream prediction tasks. TVineSynth introduces a targeted bias into the vine copula model that, combined with the specific tree structure of the vine, causes the model to zero out privacy-leaking dependencies while relying on those that are beneficial for utility. Privacy is here measured with membership (MIA) and attribute inference attacks (AIA). Further, we theoretically justify how the construction of TVineSynth ensures AIA privacy under a natural privacy measure for continuous sensitive attributes. When compared to competitor models, with and without DP, on simulated and on real-world data, TVineSynth achieves a superior privacy-utility balance. © 2025, CC BY.

关键词： Differential privacy

来源：评论

学校读者我要写书评

暂无评论

The map equation goes neural: mapping network flows with graph neural networks 24

The map equation goes neural: mapping network flows with gra...

引用

Proceedings of the 38th International Conference on Neural Information Processing Systems

作者： Christopher Blöcker Chester Tan Ingo Scholtes Data Analytics Group Department of Informatics University of Zurich Switzerland Chair of Machine Learning for Complex Networks Center for Artificial Intelligence and Data Science Julius-Maximilians-Universität Würzburg Germany

ISBN: (纸本)9798331314385

Community detection is an essential tool for unsupervised data exploration and revealing the organisational structure of networked systems. With a long history in network science, community detection typically relies on objective functions, optimised with custom-tailored search algorithms, but often without leveraging recent advances in deep learning. Recently, first works have started incorporating such objectives into loss functions for deep graph clustering and pooling. We consider the map equation, a popular information-theoretic objective function for unsupervised community detection, and express it in differentiable tensor form for optimisation through gradient descent. Our formulation turns the map equation compatible with any neural network architecture, enables end-to-end learning, incorporates node features, and chooses the optimal number of clusters automatically, all without requiring explicit regularisation. Applied to unsupervised graph clustering tasks, we achieve competitive performance against state-of-the-art deep graph clustering baselines in synthetic and real-world datasets.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Probabilistic Decomposed Linear Dynamical Systems for Robust Discovery of Latent Neural Dynamics 38

Probabilistic Decomposed Linear Dynamical Systems for Robust...

引用

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Chen, Yenho Mudrik, Noga Johnsen, Kyle A. Alagapan, Sankaraleengam Charles, Adam S. Rozell, Christopher J. Machine Learning Center Georgia Institute of Technology United States School of Electrical and Computer Engineering Georgia Institute of Technology United States Coulter Dept. of Biomedical Engineering Emory University Georgia Institute of Technology United States Department of Biomedical Engineering Mathematical Institute for Data Science Center for Imaging Science Kavli Neuroscience Discovery Institute Johns Hopkins University United States

Time-varying linear state-space models are powerful tools for obtaining mathematically interpretable representations of neural signals. For example, switching and decomposed models describe complex systems using latent variables that evolve according to simple locally linear dynamics. However, existing methods for latent variable estimation are not robust to dynamical noise and system nonlinearity due to noise-sensitive inference procedures and limited model formulations. This can lead to inconsistent results on signals with similar dynamics, limiting the model's ability to provide scientific insight. In this work, we address these limitations and propose a probabilistic approach to latent variable estimation in decomposed models that improves robustness against dynamical noise. Additionally, we introduce an extended latent dynamics model to improve robustness against system nonlinearities. We evaluate our approach on several synthetic dynamical systems, including an empirically-derived brain-computer interface experiment, and demonstrate more accurate latent variable inference in nonlinear systems with diverse noise conditions. Furthermore, we apply our method to a real-world clinical neurophysiology dataset, illustrating the ability to identify interpretable and coherent structure where previous models cannot. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Log barriers for safe black-box optimization with application to safe reinforcement learning

The Journal of Machine Learning Research

引用

The Journal of machine learning Research 2024年第1期25卷 8101-8154页

作者： Ilnura Usmanova Yarden As Maryam Kamgarpou Andreas Krause Swiss Data Science Center Paul Scherrer Institute Villigen Switzerland Institute for Machine Learning D-INFK ETH Zürich Zurich Switzerland STI-IGM-Sycamore EPFL Lausanne Switzerland

Optimizing noisy functions online, when evaluating the objective requires experiments on a deployed system, is a crucial task arising in manufacturing, robotics and various other domains. Often, constraints on safe inputs are unknown ahead of time, and we only obtain noisy information, indicating how close we are to violating the constraints. Yet, safety must be guaranteed at all times, not only for the final output of the *** introduce a general approach for seeking a stationary point in high dimensional non-linear stochastic optimization problems in which maintaining safety during learning is crucial. Our approach called LB-SGD, is based on applying stochastic gradient descent (SGD) with a carefully chosen adaptive step size to a logarithmic barrier approximation of the original problem. We provide a complete convergence analysis of non-convex, convex, and strongly-convex smooth constrained problems, with first-order and zeroth-order feedback. Our approach yields efficient updates and scales better with dimensionality compared to existing *** empirically compare the sample complexity and the computational cost of our method with existing safe learning approaches. Beyond synthetic benchmarks, we demonstrate the effectiveness of our approach on minimizing constraint violation in policy search tasks in safe reinforcement learning (RL).

关键词： stochastic optimization safe learning black-box optimization smooth constrained optimization reinforcement learning

来源：评论

学校读者我要写书评

暂无评论

Federated learning in Medical Imaging: Part II: Methods, Challenges, and Considerations

引用

Journal of the American College of Radiology 2022年第8期19卷 975-982页

作者： Darzidehkalani, Erfan Ghasemi-rad, Mohammad van Ooijen, P.M.A. Department of Radiation Oncology University Medical Center Groningen University of Groningen Groningen Netherlands Machine Learning Lab Data Science Center in Health University Medical Center Groningen University of Groningen Netherlands Department of Interventional Radiology Baylor College of Medicine Houston Texas

Federated learning is a machine learning method that allows decentralized training of deep neural networks among multiple clients while preserving the privacy of each client's data. Federated learning is instrumental in medical imaging because of the privacy considerations of medical data. Setting up federated networks in hospitals comes with unique challenges, primarily because medical imaging data and federated learning algorithms each have their own set of distinct characteristics. This article introduces federated learning algorithms in medical imaging and discusses technical challenges and considerations of real-world implementation of them. © 2022 American College of Radiology

关键词： Federated learning medical imaging privacy-preserving machine learning

来源：评论

学校读者我要写书评

暂无评论

Task Specific Pretraining with Noisy Labels for Remote Sensing Image Segmentation

Task Specific Pretraining with Noisy Labels for Remote Sensi...

引用

IEEE International Symposium on Geoscience and Remote Sensing (IGARSS)

作者： Chenying Liu Conrad M Albrecht Yi Wang Xiao Xiang Zhu Data Science in Earth Observation Technical University of Munich Germany German Aerospace Center Remote Sensing Technology Institute Germany Munich Center for Machine Learning (MCML) Germany

ISBN: (数字)9798350360325

ISBN: (纸本)9798350360332

Compared to supervised deep learning, self-supervision provides remote sensing a tool to reduce the amount of exact, human-crafted geospatial annotations. While image-level information for unsupervised pretraining efficiently works for various classification downstream tasks, the performance on pixel-level semantic segmentation lags behind in terms of model accuracy. On the contrary, many easily available label sources (e.g., automatic labeling tools and land cover land use products) exist, which can provide a large amount of noisy labels for segmentation model training. In this work, we propose to exploit noisy semantic segmentation maps for model pretraining. Our experiments provide insights on robustness per network layer. The transfer learning settings test the cases when the pretrained encoders are fine-tuned for different label classes and decoders. The results from two datasets indicate the effectiveness of task-specific supervised pretraining with noisy labels. Our findings pave new avenues to improved model accuracy and novel pretraining strategies for efficient remote sensing image segmentation.

关键词： Training Accuracy Semantic segmentation Transfer learning Land surface Robustness Noise measurement

来源：评论

学校读者我要写书评

暂无评论

HTC-DC Net: Monocular Height Estimation From Single Remote Sensing Images

引用

IEEE Transactions on Geoscience and Remote Sensing 2023年 61卷

作者： Chen, Sining Shi, Yilei Xiong, Zhitong Zhu, Xiao Xiang Chair of Data Science in Earth Observation Munich80333 Germany School of Engineering and Design Munich80333 Germany Chair of Data Science in Earth Observation The Munich Center for Machine Learning Munich80333 Germany

Three-dimensional geoinformation is of great significance for understanding the living environment;however, 3-D perception from remote sensing data, especially on a large scale, is restricted, mainly due to the high costs of 3-D sensors such as light detection and ranging (LiDAR). To tackle this problem, we propose a method for monocular height estimation from optical imagery, which is currently one of the richest sources of remote sensing data. As an ill-posed problem, monocular height estimation requires well-designed networks for enhanced representations to improve the performance. Moreover, the distribution of height values is long-tailed with the low-height pixels, e.g., the background (BG), as the head, and thus, trained networks are usually biased and tend to underestimate building heights. To solve the problems, instead of formalizing the problem as a regression task, we propose HTC-DC Net following the classification-regression paradigm, with the head-tail cut (HTC) and the distribution-based constraints (DCs) as the main contributions. HTC-DC Net is composed of the backbone network as the feature extractor, the HTC-AdaBins module, and the hybrid regression process. The HTC-AdaBins module serves as the classification phase to determine bins adaptive to each input image. It is equipped with a vision transformer (ViT) encoder to incorporate local context with holistic information and involves an HTC to address the long-tailed problem in monocular height estimation for balancing the performances of foreground (FG) and BG pixels. The hybrid regression process does the regression via the smoothing of bins from the classification phase, which is trained via DCs. The proposed network is tested on three datasets of different resolutions, namely ISPRS Vaihingen (0.09 m), data Fusion Contest 19 (DFC19) (1.3 m), and Global Building Height (GBH) (3 m). The experimental results show the superiority of the proposed network over existing methods by large margins. Extensiv

关键词： Optical radar

来源：评论

学校读者我要写书评

暂无评论

Position: insights from survey methodology can improve training data 24

Position: insights from survey methodology can improve train...

引用

Proceedings of the 41st International Conference on machine learning

作者： Stephanie Eckman Barbara Plank Frauke Kreuter Social Data Science Center University of Maryland College Park MD Center for Information and Language Processing (CIS) LMU Munich Germany and Computer Science Department IT University of Copenhagen Denmark and Munich Center for Machine Learning (MCML) LMU Munich Germany Institute for Statistics and Munich Center for Machine Learning (MCML) LMU Munich Germany and Social Data Science Center and Joint Program in Survey Methodology University of Maryland College Park MD

Whether future AI models are fair, trustworthy, and aligned with the public's interests rests in part on our ability to collect accurate data about what we want the models to do. However, collecting high-quality data is difficult, and few AI/ML researchers are trained in data collection methods. Recent research in data-centric AI has show that higher quality training data leads to better performing models, making this the right moment to introduce AI/ML researchers to the field of survey methodology, the science of data collection. We summarize insights from the survey methodology literature and discuss how they can improve the quality of training and feedback data. We also suggest collaborative research ideas into how biases in data collection can be mitigated, making models more accurate and human-centric.

关键词：

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：