检索结果-内蒙古大学图书馆

Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Shi, Yubin Chen, Yixuan Dong, Mingzhi Yang, Xiaochen Li, Dongsheng Wang, Yujiang Dick, Robert Lv, Qin Zhao, Yingying Yang, Fan Lu, Tun Gu, Ning Shang, Li China and Shanghai Key Laboratory of Data Science School of Computer Science Fudan University China School of Mathematics Statistics University of Glasgow United Kingdom Microsoft Research Asia Shanghai China Department of Engineering Science University of Oxford United Kingdom Department of Electrical Engineering and Computer Science University of Michigan United States Department of Computer Science University of Colorado Boulder United States School of Microelectronics Fudan University China

Despite their prevalence in deep-learning communities, over-parameterized models convey high demands of computational costs for proper training. This work studies the fine-grained, modular-level learning dynamics of over-parameterized models to attain a more efficient and fruitful training strategy. Empirical evidence reveals that when scaling down into network modules, such as heads in self-attention models, we can observe varying learning patterns implicitly associated with each module's trainability. To describe such modular-level learning capabilities, we introduce a novel concept dubbed modular neural tangent kernel (mNTK), and we demonstrate that the quality of a module's learning is tightly associated with its mNTK's principal eigenvalue λmax. A large λmax indicates that the module learns features with better convergence, while those miniature ones may impact generalization negatively. Inspired by the discovery, we propose a novel training strategy termed Modular Adaptive Training (MAT) to update those modules with their λmax exceeding a dynamic threshold selectively, concentrating the model on learning common features and ignoring those superfluous ones. Unlike most existing training schemes with a complete BP cycle across all network modules, MAT can significantly save computations by its partially-updating strategy and can further improve performance. Experiments show that MAT nearly halves the computational cost of model training and outperforms the accuracy of baselines. © 2024, CC BY.

关键词： Eigenvalues and eigenfunctions

Multi-Unit Floor Plan Recognition and Reconstruction Using Improved Semantic Segmentation of Raster-Wise Floor Plans

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Kratochvila, Lukas de Jong, Gijs Arkesteijn, Monique Bilík, Šimon Zemčík, Tomáš Horak, Karel Rellermeyer, Jan S. Department of Control and Instrumentation Faculty of Electrical Engineering and Communication Brno University of Technology Brno Czech Republic Department of Software Technology Faculty of Electrical Engineering Mathematics and Computer Science TU Delft Delft Netherlands Department of Management in the Built Environment Faculty of Architecture and the Built Environment TU Delft Delft Netherlands Computer Vision and Pattern Recognition Laboratory Department of Computational Engineering Lappeenranta-Lahti University of Technology LUT Lappeenranta Finland Dependable and Scalable Software Systems Institute of Systems Engineering Faculty of Electrical Engineering and Computer Science Leibniz University Hannover Hannover Germany

Digital twins have a major potential to form a significant part of urban management in emergency planning, as they allow more efficient designing of the escape routes, better orientation in exceptional situations, and faster rescue intervention. Nevertheless, creating the twins still remains a largely manual effort, due to a lack of 3D-representations, which are available only in limited amounts for some new buildings. Thus, in this paper we aim to synthesize 3D information from commonly available 2D architectural floor plans. We propose two novel pixel-wise segmentation methods based on the MDA-Unet and MACU-Net architectures with improved skip connections, an attention mechanism, and a training objective together with a reconstruction part of the pipeline, which vectorizes the segmented plans to create a 3D model. The proposed methods are compared with two other state-of-the-art techniques and several benchmark datasets. On the commonly used CubiCasa benchmark dataset, our methods have achieved the mean F1 score of 0.86 over five examined classes, outperforming the other pixel-wise approaches tested. We have also made our code publicly available to support research in the field. © 2024, CC BY.

关键词： Semantic Segmentation

Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Chen, Yineng Li, Zuchao Zhang, Lefei Du, Bo Zhao, Hai National Engineering Research Center for Multimedia Software School of Computer Science Wuhan University Wuhan430072 China Hubei Luojia Laboratory Wuhan430072 China Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai200240 China

Optimizer is an essential component for the success of deep learning, which guides the neural network to update the parameters according to the loss on the training set. SGD and Adam are two classical and effective optimizers on which researchers have proposed many variants, such as SGDM and RAdam. In this paper, we innovatively combine the backward-looking and forward-looking aspects of the optimizer algorithm and propose a novel ADMETA (A Double exponential Moving averagE To Adaptive and non-adaptive momentum) optimizer framework. For backward-looking part, we propose a DEMA variant scheme, which is motivated by a metric in the stock market, to replace the common exponential moving average scheme. While in the forward-looking part, we present a dynamic lookahead strategy which asymptotically approaches a set value, maintaining its speed at early stage and high convergence performance at final stage. Based on this idea, we provide two optimizer implementations, ADMETAR and ADMETAS, the former based on RAdam and the latter based on SGDM. Through extensive experiments on diverse tasks, we find that the proposed ADMETA optimizer outperforms our base optimizers and shows advantages over recently proposed com- Copyright © 2023, The Authors. All rights reserved.

关键词： Deep learning

Beyond Entanglement Routing: Source Assignment and All-Optical Switching-Based Distribution

学校读者我要写书评

暂无评论

IEEE/ACM Transactions on Networking 2024年

作者： Zhu, Qiucheng Zhao, Yangming Xu, Hongli Huang, Liusheng Qiao, Chunming State Key Laboratory for Novel Software Technology Nanjing University Nanjing210008 China School of Intelligent Software and Engineering Nanjing University Suzhou Campus Suzhou215163 China School of Computer Science and Technology University of Science and Technology of China Hefei230026 China Suzhou Institute for Advanced Research University of Science and Technology of China Suzhou215123 China Hefei National Laboratory University of Science and Technology of China Hefei230088 China Department of Computer Science and Engineering University at Buffalo State University of New York BuffaloNY14260 United States

Entanglement routing plays a vital role in distributed quantum computing and quantum networks. Previous works on entanglement routing have addressed several design challenges due to limited quantum resources, failures to establish entanglement, and decoherence of established entanglement, but considered neither the limitations imposed by having a limited number of entangled photon sources (EPSes), nor the benefits of using all-optical (or quantum) switching to distribute entangled photons. In this paper, we first explore the problem of jointly optimizing entanglement routing and EPS assignment, assuming no all-optical switching capability. In other words, a pair of entangled photons generated by one EPS can be distributed to two neighboring quantum nodes. We then relax the above assumption so as to allow a pair of entangled photons generated by one EPS to be distributed to non-adjacent nodes using all-optical switching. We propose two corresponding solutions, namely, entanglement routing and EPS assignment (or ERSA), and ERSAwith all-optical switching-based distribution (or ERSA+D) that aim to maximize the number of entanglement connections between the given set of source-destination (SD) pairs while avoiding starvation and achieving fairness among the SD pairs. In order to obtain efficient solutions in a large discrete solution space in a timely manner, we first formulate each optimization problem as an Integer Linear Programming (ILP), and then propose efficient algorithms to derive near-optimal solutions based on relaxation, Lagrangian decomposition, duality iteration, and rounding techniques. Extensive simulations show that ERSA+Dcan increase network throughput by up to 1652% and 113%, respectively, thanks to EPS assignment optimization, and all-optical switching based entanglement distribution. © 1993-2012 IEEE.

关键词： Quantum computers

A Robust Slot Filling Model based on LSTM and CRF for IoT Voice Interaction

学校读者我要写书评

暂无评论

A Robust Slot Filling Model based on LSTM and CRF for IoT Vo...

2022 IEEE GLOBECOM Workshops, GC Wkshps 2022

作者： Jbene, Mourad Tigani, Smail Saadane, Rachid Chehri, Abdellah Laboratory Engineering system SIRC/LAGeS-EHTP Hassania School of Public Works Casablanca Morocco Euro-Med University Fes 51 Engineering Unit Euro-Med Research Center Morocco Royal Military College of Canada Department of Mathematics and Computer Science Kingston Canada

ISBN: (纸本)9781665459754

The Internet Of Things (IoT) is an emerging paradigm representing a network of infrastructure connecting different types of devices in different places. Recently, IoT-based Conversational systems gained more attention, providing users with a new human-computer interaction mode. These new systems allow users to interact with the IoT ecosystem through a virtual assistant. They also heavily rely on Natural Language Processing (NLP) techniques for intelligent and efficient communication. Understanding the user's needs is the first step toward a more humanized IoT interaction system. This can be achieved using Natural language understanding (NLU) modules. Slot Filling is one of the core sub-tasks in NLU. It is an active research area focusing on extracting attribute values from the user's utterance. In this study, we propose an LSTM-based model for slot filling. The model benefits from contextual embeddings extracted from the BERT transformer model, the attention mechanism, and the conditional random field (CRF) that is known to model strong dependence among adjacent tags in the output sequence. Experiments show that our model achieved competitive results on two widely used benchmark datasets. © 2022 IEEE.

关键词： Internet of things

Realizing Emotional Interactions to Learn User Experience and Guide Energy Optimization for Mobile Architectures 22

学校读者我要写书评

暂无评论

Realizing Emotional Interactions to Learn User Experience an...

Proceedings of the 55th Annual IEEE/ACM International Symposium on Microarchitecture

作者： Xueliang Li Zhuobin Shi Junyang Chen Yepang Liu National Engineering Laboratory for Big Data System Computing Technology Shenzhen University China College of Computer Science and Software Engineering Shenzhen University China Research Institute of Trustworthy Autonomous Systems Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation and Department of Computer Science and Engineering Southern University of Science and Technology China

ISBN: (纸本)9781665462723

In the age of AI, mobile architectures such as smartphones are still "cold machines"; machines do not feel. If the architecture is able to feel users' feelings and runtime user experience (UX), it will accordingly adapt performance/energy to find the optimal system-operating state that consumes the least energy to satisfy users. In this paper, we will utilize users' facial expressions (FEs) to learn their runtime UX. We know that FEs are the natural and direct way for humans to convey their emotions and feelings. Our study reveals that FEs also reflect UX. Our research for the first time quantifies the link between FEs and UX. Leveraging this link, the architecture will be able to use the front camera to see FEs and feel users' UX. Based on UX, the architecture can appropriately provision computing resources. We propose Vi-energy system to realize the above idea. Our evaluation shows that Vi-energy reduces energy consumption by 52.9% at maximum and secures UX.

关键词：

On the Noise Robustness of In-Context Learning for Text Generation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Gao, Hongfu Zhang, Feipeng Jiang, Wenyu Shu, Jun Zheng, Feng Wei, Hongxin Department of Statistics and Data Science Southern University of Science and Technology China School of Economics and Finance Xi’an Jiaotong University China National Key Laboratory for Novel Software Technology Nanjing University China School of Mathematics and Statistics Xi’an Jiaotong University China Department of Computer Science and Engineering Southern University of Science and Technology China

Large language models (LLMs) have shown impressive performance on downstream tasks by in-context learning (ICL), which heavily relies on the quality of demonstrations selected from a large set of annotated examples. Recent works claim that in-context learning is robust to noisy demonstrations in text classification. In this work, we show that, on text generation tasks, noisy annotations significantly hurt the performance of in-context learning. To circumvent the issue, we propose a simple and effective approach called Local Perplexity Ranking (LPR), which replaces the "noisy" candidates with their nearest neighbors that are more likely to be clean. Our method is motivated by analyzing the perplexity deviation caused by noisy labels and decomposing perplexity into inherent perplexity and matching perplexity. Our key idea behind LPR is thus to decouple the matching perplexity by performing the ranking among the neighbors in semantic space. Our approach can prevent the selected demonstrations from including mismatched input-label pairs while preserving the effectiveness of the original selection methods. Extensive experiments demonstrate the effectiveness of LPR, improving the EM score by up to 18.75 on common benchmarks with noisy annotations. Our code is available at https://***/ml-stat-Sustech/Local-Perplexity-Ranking © 2024, CC0.

关键词： Semantics

JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Cao, Jialun Chen, Zhiyong Wu, Jiarong Cheung, Shing-Chi Xu, Chang Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong Guangzhou HKUST Fok Ying Tung Research Institute Guangzhou China State Key Laboratory for Novel Software Technology Nanjing University Nanjing China

Code generation benchmarks such as HumanEval are widely adopted to evaluate LLMs’ capabilities. However, after consolidating the latest 24 benchmarks, we noticed three significant imbalances. First, imbalanced programming language. 95.8% of benchmarks involve Python, while only 5 benchmarks involve Java, resulting in an insufficient understanding of LLMs’ capability to generate Java code. Second, imbalanced code granularity. Function-/statement-level benchmarks account for over 83.3% of benchmarks. Only a mere handful extends to class-/project-levels, and all are limited to Python. Third, lacking advanced features. Existing benchmarks primarily assess basic coding skills (e.g., variables, operators, and control structures), while overlooking advanced Object-Oriented Programming (OOP) features (i.e., encapsulation, inheritance, and polymorphism). Considering the prevalence of these advanced features in real-world Java project development, constructing benchmarks to test LLMs on handling OOP features is necessary. To fill these gaps, we propose JavaBench, a project-level Java benchmark that exercises OOP features. It comprises four Java projects with 389 methods in 106 Java classes. The test coverage is up to 92%, and JavaBench is attested by 282 undergraduate students, reaching a 90.93/100 average score (i.e., pass rate against the test suite), ensuring the quality of documentation, code skeleton, and tests. To better evaluate LLM’s capability against JavaBench, we introduce a systematic evaluation design covering three context settings and five synthesis strategies at two granularities using three hierarchical metrics. Our extensive experiment yields several interesting findings. First, we noticed that regarding project-level Java programming, LLMs are far behind undergraduate students (no project can be correctly completed by any studied LLMs, and at most 41.17% Pass@5 in a more relaxed evaluation). Second, using method signature as prompt context may strike an ide

关键词： Students

How to Design a R-D Model for UAV Video Transmission? 15

学校读者我要写书评

暂无评论

How to Design a R-D Model for UAV Video Transmission?

15th IEEE International Conference on Wireless Communications and Signal Processing, WCSP 2023

作者： Yu, Zongyang Yang, Peng Cao, Xianbin Zheng, Dezhi Quek, Tony Q. S. Oliver Wu, Dapeng Beihang University School of Electronic Information Engineering Beijing China Peng Cheng Laboratory Department of Mathematics and Theories Shenzhen China Beijing Institute of Technology Advanced Research Institute of Multidisciplinary Sciences Beijing China Singapore University of Technology and Design Information Systems Technology and Design Singapore City University of Hong Kong Department of Computer Science Kowloon Hong Kong

ISBN: (纸本)9798350324662

Unmanned aerial vehicle (UAV) video transmission has been extensively applied in many crucial fields. However, the problem of designing a rate-distortion (R-D) model for UAV video transmission, which is essential for theoretical analysis of video coding and optimization of video transmission quality, is under-studied. The designed R-D model is desired to be simple, accurate, and generic owing to the limited capability of the UAV. Observing the key role of transformed residuals in the R-D model, this paper elaborately discusses the modeling of the transformed residual distribution and proposes an estimation algorithm to estimate the statistical parameter of the distribution. Specifically, considering the stringent requirements for low complexity and high generalization capability, we first design a linear parameter estimator by mining and analyzing UAV video statistical characteristics. Further, we develop a bias update scheme to improve the accuracy of the estimator. Test results on multiple real video sequences taken by UAVs show that the proposed estimation algorithm is more accurate than benchmarks. © 2023 IEEE.

关键词： expectation maximization algorithm R-D model rate control video coding Video transmission