检索结果-内蒙古大学图书馆

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ghiasvand, Sajjad Yang, Yifan Xue, Zhiyu Alizadeh, Mahnoosh Zhang, Zheng Pedarsani, Ramtin Electrical and Computer Engineering Department UC Santa Barbara Santa BarbaraCA United States Computer Science Department UC Santa Barbara Santa BarbaraCA United States

Parameter-efficient fine-tuning (PEFT) methods typically assume that Large Language Models (LLMs) are trained on data from a single device or client. However, real-world scenarios often require fine-tuning these models on private data distributed across multiple devices. Federated Learning (FL) offers an appealing solution by preserving user privacy, as sensitive data remains on local devices during training. Nonetheless, integrating PEFT methods into FL introduces two main challenges: communication overhead and data heterogeneity. In this paper, we introduce FedTT and FedTT+, methods for adapting LLMs by integrating tensorized adapters into client-side models’ encoder/decoder blocks. FedTT is versatile and can be applied to both cross-silo FL and large-scale cross-device FL. FedTT+, an extension of FedTT tailored for cross-silo FL, enhances robustness against data heterogeneity by adaptively freezing portions of tensor factors, further reducing the number of trainable parameters. Experiments on BERT and LLaMA models demonstrate that our proposed methods successfully address data heterogeneity challenges and perform on par or even better than existing federated PEFT approaches while achieving up to 10× reduction in communication cost. © 2024, CC BY.

关键词： Data assimilation

Predicting Trust In Autonomous Vehicles: Modeling Young Adult Psychosocial Traits, Risk-Benefit Attitudes, And Driving Factors With Machine Learning 25

学校读者我要写书评

暂无评论

Predicting Trust In Autonomous Vehicles: Modeling Young Adul...

2025 CHI Conference on Human Factors in Computing Systems, CHI 2025

作者： Kaufman, Robert A Lee, Emi Bedmutha, Manas Satish Kirsh, David Weibel, Nadir Cognitive Science University of California La Jolla San Diego CA United States Uc San Diego La Jolla CA United States Cognitive Science University of California San Diego San Diego CA United States Computer Science and Engineering & the Design Lab Uc San Diego La Jolla CA United States

ISBN: (纸本)9798400713941

Low trust remains a significant barrier to Autonomous Vehicle (AV) adoption. To design trustworthy AVs, we need to better understand the individual traits, attitudes, and experiences that impact people's trust judgements. We use machine learning to understand the most important factors that contribute to young adult trust based on a comprehensive set of personal factors gathered via survey (n = 1457). Factors ranged from psychosocial and cognitive attributes to driving style, experiences, and perceived AV risks and benefits. Using the explainable AI technique SHAP, we found that perceptions of AV risks and benefits, attitudes toward feasibility and usability, institutional trust, prior experience, and a person's mental model are the most important predictors. Surprisingly, psychosocial and many technology- and driving-specific factors were not strong predictors. Results highlight the importance of individual differences for designing trustworthy AVs for diverse groups and lead to key implications for future design and research. © 2025 Copyright held by the owner/author(s).

关键词： Autonomous Vehicles Machine Learning Personalization

Joint Diffusion Processes as an Inductive Bias in Sheaf Neural Networks 1

学校读者我要写书评

暂无评论

Joint Diffusion Processes as an Inductive Bias in Sheaf Neur...

1st Geometry-Grounded Representation Learning and Generative Modeling Workshop, GRaM 2024 at the 41st International Conference on Machine Learning, ICML 2024

作者： Caralt, Ferran Hernandez Gil, Guillermo Bernárdez Duta, Iulia Liò, Pietro Cot, Eduard Alarcón FME FIB ETSETB Universitat Politècnica de Catalunya-BarcelonaTech Barcelona Spain Department of Electrical and Computer Engineering UC Santa Barbara Santa Barbara United States Department of Computer Science and Technology University of Cambridge Cambridge United Kingdom Department of Electronic Engineering Universitat Politècnica de Catalunya-BarcelonaTech Barcelona Spain

Sheaf Neural Networks (SNNs) naturally extend Graph Neural Networks (GNNs) by endowing a cellular sheaf over the graph, equipping nodes and edges with vector spaces and defining linear mappings between them. While the attached geometric structure has proven to be useful in analyzing heterophily and oversmoothing, so far the methods by which the sheaf is computed do not always guarantee a good performance in such settings. In this work, drawing inspiration from opinion dynamics concepts, we propose two novel sheaf learning approaches that (i) provide a more intuitive understanding of the involved structure maps, (ii) introduce a useful inductive bias for heterophily and oversmoothing, and (iii) infer the sheaf in a way that does not scale with the number of features, thus using fewer learnable parameters than existing methods. In our evaluation, we show the limitations of the real-world benchmarks used so far on SNNs, and design a new synthetic task–leveraging the symmetries of n-dimensional ellipsoids–that enables us to better assess the strengths and weaknesses of sheaf-based models. Our extensive experimentation on these novel datasets reveals valuable insights into the scenarios and contexts where basic SNNs and our proposed approaches can be beneficial. Copyright 2024 by the author(s).

关键词： Vector spaces

Optimal PAC Bounds Without Uniform Convergence

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Aden-Ali, Ishaq Cherapanamjeri, Yeshwanth Shetty, Abhishek Zhivotovskiy, Nikita Department of Electrical Engineering and Computer Science UC Berkeley United States Department of Statistics UC Berkeley United States

In statistical learning theory, determining the sample complexity of realizable binary classification for VC classes was a long-standing open problem. The results of Simon [Sim15] and Hanneke [Han16a] established sharp upper bounds in this setting. However, the reliance of their argument on the uniform convergence principle limits its applicability to more general learning settings such as multiclass classification. In this paper, we address this issue by providing optimal high probability risk bounds through a framework that surpasses the limitations of uniform convergence arguments. Our framework converts the leave-one-out error of permutation invariant predictors into high probability risk bounds. As an application, by adapting the one-inclusion graph algorithm of Haussler, Littlestone, and Warmuth [HLW94], we propose an algorithm that achieves an optimal PAC bound for binary classification. Specifically, our result shows that certain aggregations of one-inclusion graph algorithms are optimal, addressing a variant of a classic question posed by Warmuth [War04]. We further instantiate our framework in three settings where uniform convergence is provably suboptimal. For multiclass classification, we prove an optimal risk bound that scales with the one-inclusion hypergraph density of the class, addressing the suboptimality of the analysis of Daniely and Shalev-Shwartz [DS14]. For partial hypothesis classification, we determine the optimal sample complexity bound, resolving a question posed by Alon, Hanneke, Holzman, and Moran [AHHM22]. For realizable bounded regression with absolute loss, we derive an optimal risk bound that relies on a modified version of the scale-sensitive dimension, refining the results of Bartlett and Long [BL98]. Our rates surpass standard uniform convergence-based results due to the smaller complexity measure in our risk bound. © 2023, CC BY.

关键词： Classification (of information)

Coprocessor Actor Critic: A Model-Based Reinforcement Learning Approach For Adaptive Brain Stimulation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Pan, Michelle Schrum, Mariah Myers, Vivek Bıyık, Erdem Dragan, Anca Department of Electrical Engineering and Computer Sciences UC Berkeley United States Department of Computer Science University of Southern California United States

Adaptive brain stimulation can treat neurological conditions such as Parkinson’s disease and post-stroke motor deficits by influencing abnormal neural activity. Because of patient heterogeneity, each patient requires a unique stimulation policy to achieve optimal neural responses. Model-free reinforcement learning (MFRL) holds promise in learning effective policies for a variety of similar control tasks, but is limited in domains like brain stimulation by a need for numerous costly environment interactions. In this work we introduce Coprocessor Actor Critic, a novel, model-based reinforcement learning (MBRL) approach for learning neural coprocessor policies for brain stimulation. Our key insight is that coprocessor policy learning is a combination of learning how to act optimally in the world and learning how to induce optimal actions in the world through stimulation of an injured brain. We show that our approach overcomes the limitations of traditional MFRL methods in terms of sample efficiency and task success and outperforms baseline MBRL approaches in a neurologically realistic model of an injured brain. © 2024, CC BY.

关键词： Reinforcement learning

On Emergence of Clean-Priority Learning in Early Stopped Neural Networks

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Liu, Chaoyue Abedsoltan, Amirhesam Belkin, Mikhail Halicioglu Data Science Institute UC San Diego United States Department of Computer Science and Engineering

When random label noise is added to a training dataset, the prediction error of a neural network on a label-noise-free test dataset initially improves during early training but eventually deteriorates, following a U-shaped dependence on training time. This behaviour is believed to be a result of neural networks learning the pattern of clean data first and fitting the noise later in the training, a phenomenon that we refer to as clean-priority learning. In this study, we aim to explore the learning dynamics underlying this phenomenon. We theoretically demonstrate that, in the early stage of training, the update direction of gradient descent is determined by the clean subset of training data, leaving the noisy subset has minimal to no impact, resulting in a prioritization of clean learning. Moreover, we show both theoretically and experimentally, as the clean-priority learning goes on, the dominance of the gradients of clean samples over those of noisy samples diminishes, and finally results in a termination of the clean-priority learning and fitting of the noisy samples. Copyright © 2023, The Authors. All rights reserved.

关键词： Gradient methods

Save It for the "Hot" Day: An LLM-Empowered Visual Analytics System for Heat Risk Management

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Haobo Kam-Kwai, Wong Luo, Yan Chen, Juntong Liu, Chengzhong Zhang, Yaxuan Lau, Alexis Kai Hon Qu, Huamin Liu, Dongyu Department of Computer Science and Engineering HKUST Hong Kong School of Computer Science and Technology ECNU China Division of Environment and Sustainability HKUST Hong Kong Department of Computer Science UC Davis United States

The escalating frequency and intensity of heat-related climate events, particularly heatwaves, emphasize the pressing need for advanced heat risk management strategies. Current approaches, primarily relying on numerical models, face challenges in spatial-temporal resolution and in capturing the dynamic interplay of environmental, social, and behavioral factors affecting heat risks. This has led to difficulties in translating risk assessments into effective mitigation actions. Recognizing these problems, we introduce a novel approach leveraging the burgeoning capabilities of Large Language Models (LLMs) to extract rich and contextual insights from news reports. We hence propose an LLM-empowered visual analytics system, Havior, that integrates the precise, data-driven insights of numerical models with nuanced news report information. This hybrid approach enables a more comprehensive assessment of heat risks and better identification, assessment, and mitigation of heat-related threats. The system incorporates novel visualization designs, such as "thermoglyph" and news glyph, enhancing intuitive understanding and analysis of heat risks. The integration of LLM-based techniques also enables advanced information retrieval and semantic knowledge extraction that can be guided by experts’ analytics needs. Our case studies on two cities that faced significant heatwave events and interviews with five experts have demonstrated the usefulness of our system in providing in-depth and actionable insights for heat risk management. © 2024, CC BY.

关键词： Numerical models

Toward Large Kernel Models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Abedsoltan, Amirhesam Belkin, Mikhail Pandit, Parthe Department of Computer Science and Engineering Halicioglu Data Science Institute UC San Diego United States

Recent studies indicate that kernel machines can often perform similarly or better than deep neural networks (DNNs) on small datasets. The interest in kernel machines has been additionally bolstered by the discovery of their equivalence to wide neural networks in certain regimes. However, a key feature of DNNs is their ability to scale the model size and training data size independently, whereas in traditional kernel machines model size is tied to data size. Because of this coupling, scaling kernel machines to large data has been computationally challenging. In this paper, we provide a way forward for constructing large-scale general kernel models, which are a generalization of kernel machines that decouples the model and data, allowing training on large datasets. Specifically, we introduce EigenPro 3.0, an algorithm based on projected dual preconditioned SGD and show scaling to model and data sizes which have not been possible with existing kernel methods. We provide a PyTorch based implementation which can take advantage of multiple GPUs. © 2023, CC BY.

关键词： Deep neural networks

Fast multiplication of random dense matrices with sparse matrices

学校读者我要写书评

暂无评论

Fast multiplication of random dense matrices with sparse mat...

International Symposium on Parallel and Distributed Processing (IPDPS)

作者： Tianyu Liang Riley Murray Aydın Buluç James Demmel Electrical Engineering and Computer Science Department UC Berkeley Computational Research Division Lawrence Berkeley National Lab International Computer Science Institute Sandia National Laboratories

ISBN: (数字)9798350387117

ISBN: (纸本)9798350387124

This work focuses on accelerating the multiplication of a dense random matrix with a (fixed) sparse matrix, which is frequently used in sketching algorithms. We develop a novel scheme that takes advantage of blocking and recomputation (on-the-fly random number generation) to accelerate this operation. The techniques we propose decrease memory movement, thereby increasing the algorithm’s parallel scalability in shared memory architectures. On the Intel Frontera architecture, our algorithm can achieve 2x speedups over libraries such as Eigen and Intel MKL on some examples. In addition, with 32 threads, we can obtain a parallel efficiency of up to 45%.We also present a theoretical analysis for the memory movement lower bound of our algorithm, showing that under mild assumptions, it’s possible to beat the data movement lower bound of general matrix-matrix multiply (GEMM) by a factor of M, where M is the cache size. Finally, we incorporate our sketching method into a randomized algorithm for overdetermined least squares with sparse data matrices. Our results are competitive with SuiteSparse for highly overdetermined problems; in some cases, we obtain a speedup of 10x over SuiteSparse.

关键词： Distributed processing Scalability Instruction sets Memory management Memory architecture Libraries Sparse matrices