检索结果-内蒙古大学图书馆

Sharing knowledge for meta-learning with feature descriptions 22

学校读者我要写书评

暂无评论

Sharing knowledge for meta-learning with feature description...

Proceedings of the 36th International Conference on Neural Information Processing Systems

作者： Tomoharu Iwata Atsutoshi Kumagai NTT Communication Science Laboratories Kyoto Japan NTT Computer and Data Science Laboratories Tokyo Japan

ISBN: (纸本)9781713871088

Language is an important tool for humans to share knowledge. We propose a meta-learning method that shares knowledge across supervised learning tasks using feature descriptions written in natural language, which have not been used in the existing meta-learning methods. The proposed method improves the predictive performance on unseen tasks with a limited number of labeled data by meta-learning from various tasks. With the feature descriptions, we can find relationships across tasks even when their feature spaces are different. The feature descriptions are encoded using a language model pretrained with a large corpus, which enables us to incorporate human knowledge stored in the corpus into meta-learning. In our experiments, we demonstrate that the proposed method achieves better predictive performance than the existing meta-learning methods using a wide variety of real-world datasets provided by the statistical office of the EU and Japan.

关键词：

Efficient Algorithm for K-Multiple-Means

学校读者我要写书评

暂无评论

Proceedings of the ACM on Management of data 2024年第1期2卷 1-26页

作者： Yasuhiro Fujiwara Atsutoshi Kumagai Yasutoshi Ida Masahiro Nakano Makoto Nakatsuji Akisato Kimura NTT Communication Science Laboratories Atsugi-shi Kanagawa Japan NTT Computer and Data Science Laboratories Musashino-shi Tokyo Japan NTT Human Informatics Laboratories Yokosuka-shi Kanagawa Japan

K-Multiple-Means is an extension of K-means for the clustering of multiple means used in many applications, such as image segmentation, load balancing, and blind-source separation. Since K-means uses only one mean to represent each cluster, it fails to capture non-spherical cluster structures of data points. However, since K-Multiple-Means represents the cluster by computing multiple means and grouping them into specified c clusters, it can effectively capture the non-spherical clusters of the data points. To obtain the clusters, K-Multiple-Means updates a similarity matrix of a bipartite graph between the data points and the multiple means by iteratively computing the leading c singular vectors of the matrix. K-Multiple-Means, however, incurs a high computation cost for large-scale data due to the iterative SVD computations. Our proposal, F-KMM, increases the efficiency of K-Multiple-Means by computing the singular vectors from a smaller similarity matrix between the multiple means obtained from the similarity matrix of the bipartite graph. To compute the similarity matrix of the bipartite graph efficiently, we skip unnecessary distance computations and estimate lower bounding distances between the data points and the multiple means. Theoretically, the proposed approach guarantees the same clustering results as K-Multiple-Means since it can exactly compute the singular vectors from the similarity matrix between the multiple means. Experiments show that our approach is several orders of magnitude faster than previous clustering approaches that use multiple means.

关键词： algorithm clustering efficient

Adversarial Finetuning with Latent Representation Constraint to Mitigate Accuracy-Robustness Tradeoff

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Suzuki, Satoshi Yamaguchi, Shinya Takeda, Shoichiro Kanai, Sekitoshi Makishima, Naoki Ando, Atsushi Masumura, Ryo NTT Computer and Data Science Laboratories Kyoto University Japan NTT Human Informatics Laboratories

This paper addresses the tradeoff between standard accuracy on clean examples and robustness against adversarial examples in deep neural networks (DNNs). Although adversarial training (AT) improves robustness, it degrades the standard accuracy, thus yielding the tradeoff. To mitigate this tradeoff, we propose a novel AT method called ARREST, which comprises three components: (i) adversarial finetuning (AFT), (ii) representation-guided knowledge distillation (RGKD), and (iii) noisy replay (NR). AFT trains a DNN on adversarial examples by initializing its parameters with a DNN that is standardly pretrained on clean examples. RGKD and NR respectively entail a regularization term and an algorithm to preserve latent representations of clean examples during AFT. RGKD penalizes the distance between the representations of the standardly pretrained and AFT DNNs. NR switches input adversarial examples to nonadversarial ones when the representation changes significantly during AFT. By combining these components, ARREST achieves both high standard accuracy and robustness. Experimental results demonstrate that ARREST mitigates the tradeoff more effectively than previous AT-based methods do. Copyright © 2023, The Authors. All rights reserved.

关键词： Deep neural networks

Pruning Randomly Initialized Neural Networks with Iterative Randomization

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Chijiwa, Daiki Yamaguchi, Shinya Ida, Yasutoshi Umakoshi, Kenji Inoue, Tomohiro NTT Computer and Data Science Laboratories NTT Corporation Japan NTT Social Informatics Laboratories NTT Corporation Japan

Pruning the weights of randomly initialized neural networks plays an important role in the context of lottery ticket hypothesis. Ramanujan et al. [26] empirically showed that only pruning the weights can achieve remarkable performance instead of optimizing the weight values. However, to achieve the same level of performance as the weight optimization, the pruning approach requires more parameters in the networks before pruning and thus more memory space. To overcome this parameter inefficiency, we introduce a novel framework to prune randomly initialized neural networks with iteratively randomizing weight values (IteRand). Theoretically, we prove an approximation theorem in our framework, which indicates that the randomizing operations are provably effective to reduce the required number of the parameters. We also empirically demonstrate the parameter efficiency in multiple experiments on CIFAR-10 and ImageNet. The code is available at https://***/dchiji-ntt/iterand. Copyright © 2021, The Authors. All rights reserved.

关键词： Machine learning

Improving Raft Performance with Bulk Transfers

学校读者我要写书评

暂无评论

Improving Raft Performance with Bulk Transfers

International Symposium on Computing and Networking Workshops (CANDARW)

作者： Aoi Yamashita Masahiro Tanaka Yutaro Bessho Yasuhiro Fujiwara Hideyuki Kawashima Graduate School of Keio University NTT Computer and Data Science Laboratories NTT Communication Science Laboratories Keio University

Raft is known as a replicating state machine that tolerates crash faults, and its use enables various distributed systems, including distributed transaction processing systems. The original Raft protocol creates an intra-cluster consensus by having the leader generate a log of each incoming client request and replicate it to its followers. When requests arrive frequently, the number of replications increases, increasing the number of communications and possibly leading to performance degradation. To solve this problem, we propose a bulk-transfer protocol that replicates multiple requests at once. We implemented the Raft system to verify the effect of the proposed method. We measured the performance of the implemented system when multiple requests were transferred in bulk and observed an increase in throughput with an increase in the amount of data replicated at a time. The highest throughput of 214 MB/sec was achieved when replicating 12.5 MB at a time. This is 12,066 times higher than the performance when replicating one request (18 B) at a time. Besides, the size of the log is reduced to 5/9 of the conventional method.

关键词：

Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Chijiwa, Daiki Yamaguchi, Shin'ya Kumagai, Atsutoshi Ida, Yasutoshi NTT Computer and Data Science Laboratories NTT Corporation Japan Kyoto University Japan

Few-shot learning for neural networks (NNs) is an important problem that aims to train NNs with a few data. The main challenge is how to avoid overfitting since over-parameterized NNs can easily overfit to such small dataset. Previous work (e.g. MAML by Finn et al. 2017) tackles this challenge by meta-learning, which learns how to learn from a few data by using various tasks. On the other hand, one conventional approach to avoid overfitting is restricting hypothesis spaces by endowing sparse NN structures like convolution layers in computer vision. However, although such manually-designed sparse structures are sample-efficient for sufficiently large datasets, they are still insufficient for few-shot learning. Then the following questions naturally arise: (1) Can we find sparse structures effective for few-shot learning by meta-learning? (2) What benefits will it bring in terms of meta-generalization? In this work, we propose a novel meta-learning approach, called Meta-ticket, to find optimal sparse subnetworks for few-shot learning within randomly initialized NNs. We empirically validated that Meta-ticket successfully discover sparse subnetworks that can learn specialized features for each given task. Due to this task-wise adaptation ability, Meta-ticket achieves superior meta-generalization compared to MAML-based methods especially with large NNs. Copyright © 2022, The Authors. All rights reserved.

关键词： Large dataset

Effects of local minima and bifurcation delay on combinatorial optimization with continuous variables

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Sato, Shintaro NTT Computer and Data Science Laboratories NTT Corporation Musashino180-8585 Japan

Combinatorial optimization problems can be mapped onto Ising models, and their ground state is generally difficult to find. A lot of heuristics for these problems have been proposed, and one promising approach is to use continuous variables. In recent years, one such algorithm has been implemented by using parametric oscillators known as coherent Ising machines. Although these algorithms have been confirmed to have high performance through many experiments, unlike other familiar algorithms such as simulated annealing, their computational ability has not been fully investigated. In this paper, we propose a simple heuristic based on continuous variables whose static and dynamical properties are easy to investigate. Through the analyses of the proposed algorithm, we find that many local minima in the early stage of the optimization and bifurcation delay reduce its performance in a certain class of Ising models. Copyright © 2022, The Authors. All rights reserved.

关键词： Combinatorial optimization

COVARIANCE-AWARE FEATURE ALIGNMENT WITH PRE-COMPUTED SOURCE STATISTICS FOR TEST-TIME ADAPTATION TO MULTIPLE IMAGE CORRUPTIONS

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Adachi, Kazuki Yamaguchi, Shin'ya Kumagai, Atsutoshi NTT Computer and Data Science Laboratories Japan Kyoto University Japan

Real-world image recognition systems often face corrupted input images, which cause distribution shifts and degrade the performance of models. These systems often use a single prediction model in a central server and process images sent from various environments, such as cameras distributed in cities or cars. Such single models face images corrupted in heterogeneous ways in test time. Thus, they require to instantly adapt to the multiple corruptions during testing rather than being re-trained at a high cost. Test-time adaptation (TTA), which aims to adapt models without accessing the training dataset, is one of the settings that can address this problem. Existing TTA methods indeed work well on a single corruption. However, the adaptation ability is limited when multiple types of corruption occur, which is more realistic. We hypothesize this is because the distribution shift is more complicated, and the adaptation becomes more difficult in case of multiple corruptions. In fact, we experimentally found that a larger distribution gap remains after TTA. To address the distribution gap during testing, we propose a novel TTA method named Covariance-Aware Feature alignment (CAFe). We empirically show that CAFe outperforms prior TTA methods on image corruptions, including multiple types of corruptions. Copyright © 2022, The Authors. All rights reserved.

关键词： Crime

MagicPool: Dealing with Magic State Distillation Failures on Large-Scale Fault-Tolerant Quantum computer

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hirano, Yutaka Suzuki, Yasunari Fujii, Keisuke Osaka University Japan NTT Computer and Data Science Laboratories NTT Corporation Japan Japan

Magic state distillation, which is a probabilistic process used to generate magic states, plays an important role in universal fault-tolerant quantum computers. On the other hand, to solve interesting problems, we need to run complex programs on fault-tolerant quantum computers, and hence, the system needs to use hardware resources efficiently. Taking advantage of parallelism is a major optimization strategy and compilers are responsible for performing optimizations to allow parallel processing. However, the probabilistic nature of magic state distillation is not compatible with compile-time optimizations and results in an additional run-time delay. To reduce the additional run-time delay, we propose introducing a pool of magic states. We run simulations of quantum circuits to verify the magnitude of the run-time delay and the usefulness of the mitigation approach. The experimental results show that the run-time delay is amplified by parallel processing, and pooling effectively reduces the run-time delay with a small spatial cost. © 2024, CC BY.

关键词： Time delay