检索结果-内蒙古大学图书馆

INTERPRETABLE LANGUAGE MODELING VIA INDUCTION-HEAD NGRAM MODELS

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Kim, Eunji Mantena, Sriya Yang, Weiwei Singh, Chandan Yoon, Sungroh Gao, Jianfeng Microsoft Research United States Department of Electrical and Computer Engineering Seoul National University Korea Republic of Stanford University United States Interdisciplinary Program in Artificial Intelligence Seoul National University Korea Republic of

Recent large language models (LLMs) have excelled across a wide range of tasks, but their use in high-stakes and compute-limited settings has intensified the demand for interpretability and efficiency. We address this need by proposing Induction-head ngram models (Induction-Gram), a method that builds an efficient, interpretable LM by bolstering modern ngram models with a hand-engineered "induction head". This induction head uses a custom neural similarity metric to efficiently search the model’s input context for potential next-word completions. This process enables Induction-Gram to provide ngram-level grounding for each generated token. Moreover, experiments show that this simple method significantly improves next-word prediction over baseline interpretable models (up to 26%p) and can be used to speed up LLM inference for large models through speculative decoding. We further study Induction-Gram in a natural-language neuroscience setting, where the goal is to predict the next fMRI response in a sequence. It again provides a significant improvement over interpretable models (20% relative increase in the correlation of predicted fMRI responses), potentially enabling deeper scientific investigation of language selectivity in the brain. The code is available at https://***/ejkim47/induction-gram. © 2024, CC BY-NC-SA.

关键词： Digital elevation model

FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Jang, Jaehee Ha, Heonseok Jung, Dahuin Yoon, Sungroh Department of Electrical and Computer Engineering Seoul National University Seoul Korea Republic of Department of Electrical and Computer Engineering Interdisciplinary Program in Artificial Intelligence Seoul National University Seoul Korea Republic of

Personalized federated learning is aimed at allowing numerous clients to train personalized models while participating in collaborative training in a communication-efficient manner without exchanging private data. However, many personalized federated learning algorithms assume that clients have the same neural network architecture, and those for heterogeneous models remain understudied. In this study, we propose a novel personalized federated learning method called federated classifier averaging (FedClassAvg). Deep neural networks for supervised learning tasks consist of feature extractor and classifier layers. FedClassAvg aggregates classifier weights as an agreement on decision boundaries on feature spaces so that clients with not independently and identically distributed (non-iid) data can learn about scarce labels. In addition, local feature representation learning is applied to stabilize the decision boundaries and improve the local feature extraction capabilities for clients. While the existing methods require the collection of auxiliary data or model weights to generate a counterpart, FedClassAvg only requires clients to communicate with a couple of fully connected layers, which is highly communication-efficient. Moreover, FedClassAvg does not require extra optimization problems such as knowledge transfer, which requires intensive computation overhead. We evaluated FedClassAvg through extensive experiments and demonstrated it outperforms the current state-of-the-art algorithms on heterogeneous personalized federated learning tasks. © 2022, CC BY.

关键词： Deep neural networks

A Comprehensive Survey of Time Series Forecasting: Architectural Diversity and Open Challenges

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Kim, Jongseon Kim, Hyungjoon Kim, HyunGi Lee, Dongjun Yoon, Sungroh Interdisciplinary Program in Artificial Intelligence Seoul National University Korea Republic of Department of Electrical and Computer Engineering Seoul National University Korea Republic of R&D Department LG Chem R&D Department Samsung SDI

Time series forecasting is a critical task that provides key information for decision-making across various fields, such as economic planning, supply chain management, and medical diagnosis. After the use of traditional statistical methodologies and machine learning in the past, various fundamental deep learning architectures such as MLPs, CNNs, RNNs, and GNNs have been developed and applied to solve time series forecasting problems. However, the structural limitations caused by the inductive biases of each deep learning architecture constrained their performance. Transformer models, which excel at handling long-term dependencies, have become significant architectural components for time series forecasting. However, recent research has shown that alternatives such as simple linear layers can outperform Transformers. These findings have opened up new possibilities for using diverse architectures. In this context of exploration into various models, the architectural modeling of time series forecasting has now entered a renaissance. This survey not only provides a historical context for time series forecasting but also offers comprehensive and timely analysis of the movement toward architectural diversification. By comparing and re-examining various deep learning models, we uncover new perspectives and presents the latest trends in time series forecasting, including the emergence of hybrid models, diffusion models, Mamba models, and foundation models. By focusing on the inherent characteristics of time series data, we also address open challenges that have gained attention in time series forecasting, such as channel dependency, distribution shift, causality, and feature extraction. This survey explores vital elements that can enhance forecasting performance through diverse approaches. These contributions lead to lowering the entry barriers for newcomers to the field of time series forecasting, while also offering seasoned researchers broad perspectives, new opportuniti

关键词： Supply chain management

Contract-Inspired Contest Theory for Controllable Image Generation in Mobile Edge Metaverse

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Liu, Guangyuan Du, Hongyang Wang, Jiacheng Niyato, Dusit Kim, Dong In College of Computing and Data Science The Energy Research Institute @ NTU Interdisciplinary Graduate Program Nanyang Technological University Singapore Department of Electrical and Electronic Engineering University of Hong Kong Hong Kong College of Computing and Data Science Nanyang Technological University Singapore Department of Electrical and Computer Engineering Sungkyunkwan University Korea Republic of

The rapid advancement of immersive technologies has propelled the development of the Metaverse, where the convergence of virtual and physical realities necessitates the generation of high-quality, photorealistic images to enhance user experience. However, generating these images, especially through Generative Diffusion Models (GDMs), in mobile edge computing environments presents significant challenges due to the limited computing resources of edge devices and the dynamic nature of wireless networks. This paper proposes a novel framework that integrates contract-inspired contest theory, Deep Reinforcement Learning (DRL), and GDMs to optimize image generation in these resource-constrained environments. The framework addresses the critical challenges of resource allocation and semantic data transmission quality by incentivizing edge devices to efficiently transmit high-quality semantic data, which is essential for creating realistic and immersive images. The use of contest and contract theory ensures that edge devices are motivated to allocate resources effectively, while DRL dynamically adjusts to network conditions, optimizing the overall image generation process. Experimental results demonstrate that the proposed approach not only improves the quality of generated images but also achieves superior convergence speed and stability compared to traditional methods. This makes the framework particularly effective for optimizing complex resource allocation tasks in mobile edge Metaverse applications, offering enhanced performance and efficiency in creating immersive virtual environments. © 2025, CC BY.

关键词： Mobile edge computing

Effects of Incident Angle on Single-Event Leakage Current Degradation in Heavy-Ion Irradiated 1800 v and 4500 v SiC Power MOSFETs

学校读者我要写书评

暂无评论

IEEE Transactions on Nuclear Science 2025年

作者： Sengupta, A. Martinella, C. Ball, D.R. Niskanen, K. Senarath, A.S. Islam, S. Harris, P.M. Hutson, J.M. Osheroff, J.M. Sternberg, A.L. Kettunen, H. Kosier, S.L. Grossner, U. Javanainen, A. Galloway, K.F. Schrimpf, R.D. Witulski, A.F. Zürich8092 Switzerland Vanderbilt University Interdisciplinary Materials Science Program NashvilleTN37235 United States Lipscomb University Electrical and Computer Engineering Department NashvilleTN37204 United States NASA/GSFC Code 561.4 GreenbeltMD20771 United States Vanderbilt University Electrical Engineering and Computer Science Department B 351825 NashvilleTN37235 United States University of Jyväskylä Department of Physics Jyväskylä40014 Finland

Experimental heavy-ion responses of two variants of SiC power MOSFETs are evaluated. The devices have similar epitaxial thickness but different doping. The higher doping in the epitaxial region results in lower breakdown voltage, and typically leads to a lower threshold drain voltage at which single-event leakage current (SELC) occurs. Furthermore, the heavy-ion-induced SELC degradation in the drain and gate currents of the SiC MOSFETs exhibit a strong dependence on the heavy-ion's angle of incidence. The devices' behavior significantly alters when irradiated with heavy-ions at different incident angles under the same fixed bias and total fluence, as also confirmed by 3-D Technology computer aided Design (TCAD) simulations. © 1963-2012 IEEE.

关键词： Power MOSFET

Weakly Supervised Semantic Segmentation using Out-of-Distribution Data

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Lee, Jungbeom Oh, Seong Joon Yun, Sangdoo Choe, Junsuk Kim, Eunji Yoon, Sungroh Department Of Electrical And Computer Engineering Seoul National University Korea Republic of NAVER AI Lab University Of Tübingen Germany Department Of Computer Science And Engineering Sogang University Korea Republic of Interdisciplinary Program In AI AIIS ASRI INMC ISRC And NSI Seoul National University Korea Republic of

Weakly supervised semantic segmentation (WSSS) methods are often built on pixel-level localization maps obtained from a classifier. However, training on class labels only, classifiers suffer from the spurious correlation between foreground and background cues (e.g. train and rail), fundamentally bounding the performance of WSSS. There have been previous endeavors to address this issue with additional supervision. We propose a novel source of information to distinguish foreground from the background: Out-of-Distribution (OoD) data, or images devoid of foreground object classes. In particular, we utilize the hard OoDs that the classifier is likely to make false-positive predictions. These samples typically carry key visual features on the background (e.g. rail) that the classifiers often confuse as foreground (e.g. train), so these cues let classifiers correctly suppress spurious background cues. Acquiring such hard OoDs does not require an extensive amount of annotation efforts;it only incurs a few additional image-level labeling costs on top of the original efforts to collect class labels. We propose a method, W-OoD, for utilizing the hard OoDs. W-OoD achieves state-of-the-art performance on Pascal VOC 2012. The code is available at: https://***/naver-ai/w-ood. © 2022, CC BY-NC-SA.

关键词： Semantic Segmentation

Controlled Text Generation for Black-box Language Models via Score-based Progressive Editor

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yu, Sangwon Lee, Changmin Lee, Hojin Yoon, Sungroh Department of Electrical and Computer Engineering Seoul National University Korea Republic of Kakao Corp Korea Republic of Interdisciplinary Program in Artificial Intelligence Seoul National University Korea Republic of

Controlled text generation is very important for the practical use of language models because it ensures that the produced text includes only the desired attributes from a specific domain or dataset. Existing methods, however, are inapplicable to black-box models or suffer a significant trade-off between controlling the generated text and maintaining its fluency. This paper introduces the Score-based Progressive Editor (ScoPE), a novel approach designed to overcome these issues. ScoPE modifies the context at the token level during the generation process of a backbone language model. This modification guides the subsequent text to naturally include the target attributes. To facilitate this process, ScoPE employs a training objective that maximizes a target score, thoroughly considering both the ability to guide the text and its fluency. Experimental results on diverse controlled generation tasks demonstrate that ScoPE can effectively regulate the attributes of the generated text while fully utilizing the capability of the backbone large language models. Our codes are available at https://***/ysw1021/ScoPE. Copyright © 2023, The Authors. All rights reserved.

关键词： Economic and social effects

High-Resolution Hydrogen Molecule Detection Utilizing a Nanocandle-Based Optical Sensor 16

学校读者我要写书评

暂无评论

High-Resolution Hydrogen Molecule Detection Utilizing a Nano...

16th Pacific Rim Conference on Lasers and Electro-Optics, CLEO-PR 2024

作者： Park, Nu-Ri Kim, Min-Joong Lee, Jongsu Seo, Minah Ryu, Yong-Sang Kim, Myung-Ki KU-KIST Graduate School of Converging Science and Technology Korea University Seoul02841 Korea Republic of Korea Institute of Science and Technology Sensor System Research Center Seoul02792 Korea Republic of Northwestern University Department of Electrical Engineering and Computer Science EvanstonIL60208 United States School of Biomedical Engineering Korea University Seoul02481 Korea Republic of Interdisciplinary Program in Precision Public Health Korea University Seoul02481 Korea Republic of

Our research introduces a novel method for safe hydrogen detection. We've developed an advanced nano-candle sensor, combining Pd with nano-candles, enabling accurate detection of low-concentration hydrogen ( ©... 详细信息

ISBN: (纸本)9798350372076

关键词： Nanoclay

Evaluation of Chest CT-Scan Anomaly Detection Models Using Principal Component Analysis

学校读者我要写书评

暂无评论

Evaluation of Chest CT-Scan Anomaly Detection Models Using P...

Information Technology (ACIT)

作者： Ala’ Alyabrodi Shrouq Al-Daja MohammadNoor Injadat Abdallah Moubayed Ali Elrashidi Cyber Security Department Zarqa University Zarqa Jordan Data Science & AI Department Zarqa University Zarqa Jordan Computer Engineering Department and Interdisciplinary Research Center for Intelligent Secure Systems King Fahd University of Petroleum & Minerals Dhahran Saudi Arabia Department of Electrical Engineering University of Business and Technology Jeddah Saudi Arabia

ISBN: (数字)9798331540012

ISBN: (纸本)9798331540029

Machine learning-based improvements in anomaly detection, visualization, and segmentation are made possible by the growing digitization of medical imaging, which reduces the workload for medical specialists. Nevertheless, trustworthy labeled data is necessary for supervised machine learning and is sometimes expensive, time-consuming, or impossible to get. Consequently, techniques that need either little labeling (unsupervised methods) or partial labeling (semi-supervised methods) have been used increasingly been used. One approach that can be used to tackle medical imaging tasks like segmentation and classification is anomaly detection, which makes use of both unsupervised and semi-supervised methods. This study investigates the anomaly detection capabilities of deep learning and machine learning models trained on chest CT scans. To adopt this approach, firstly, by evaluating several feature extraction techniques (Principal Component Analysis (PCA), Autoencoder, and variance threshold), PCA achieves the best results on the selected dataset. After that, different machine learning and deep learning models (Isolation Forest, K-means, feed-forward neural networks, and convolutional neural networks) are assessed to detect anomalies using PCA for dimensionality reduction. The experiments show that the NN-Feedforward model with PCA outperformed the other models, also attains remarkably low false alarm rate, demonstrating its effectiveness in reducing false positives and precisely detecting anomalies.

关键词： Deep learning Image segmentation Forestry Feature extraction Feedforward neural networks Labeling Information technology Anomaly detection Principal component analysis Biomedical imaging