检索结果-内蒙古大学图书馆

Variance reduced local SGD with lower communication complexity

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Liang, Xianfeng Shen, Shuheng Liu, Jingchang Pan, Zhen Chen, Enhong Cheng, Yifei Anhui Province Key Laboratory of Big Data Analysis and Application University of Science and Technology of China Hong Kong University of Science and Technology

To accelerate the training of machine learning models, distributed stochastic gradient descent (SGD) and its variants have been widely adopted, which apply multiple workers in parallel to speed up training. Among them, Local SGD has gained much attention due to its lower communication cost. Nevertheless, when the data distribution on workers is non-identical, Local SGD requires O(T 3 4 N 3 4 ) communications to maintain its linear iteration speedup property, where T is the total number of iterations and N is the number of workers. In this paper, we propose Variance Reduced Local SGD (VRL-SGD) to further reduce the communication complexity. Benefiting from eliminating the dependency on the gradient variance among workers, we theoretically prove that VRL-SGD achieves a linear iteration speedup with a lower communication complexity O(T 1 2 N 3 2 ) even if workers access non-identical datasets. We conduct experiments on three machine learning tasks, and the experimental results demonstrate that VRL-SGD performs impressively better than Local SGD when the data among workers are quite diverse. Copyright © 2019, The Authors. All rights reserved.

关键词： Computational complexity

Fine-tuning by curriculum learning for non-autoregressive neural machine translation

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Guo, Junliang Tan, Xu Xu, Linli Qin, Tao Chen, Enhong Liu, Tie-Yan Anhui Province Key Laboratory of Big Data Analysis and Application School of Computer Science and Technology University of Science and Technology of China Microsoft Research

Non-autoregressive translation (NAT) models remove the dependence on previous target tokens and generate all target tokens in parallel, resulting in significant inference speedup but at the cost of inferior translation accuracy compared to autoregressive translation (AT) models. Considering that AT models have higher accuracy and are easier to train than NAT models, and both of them share the same model configurations, a natural idea to improve the accuracy of NAT models is to transfer a well-trained AT model to an NAT model through fine-tuning. However, since AT and NAT models differ greatly in training strategy, straightforward fine-tuning does not work well. In this work, we introduce curriculum learning into fine-tuning for NAT. Specifically, we design a curriculum in the fine-tuning process to progressively switch the training from autoregressive generation to non-autoregressive generation. Experiments on four benchmark translation datasets show that the proposed method achieves good improvement (more than 1 BLEU score) over previous NAT baselines in terms of translation accuracy, and greatly speed up (more than 10 times) the inference process over AT baselines. Copyright © 2019, The Authors. All rights reserved.

关键词： Curricula

Residual objectness for imbalance reduction

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Chen, Joya Liu, Dong Luo, Bin Peng, Xuezheng Xu, Tong Chen, Enhong Anhui Province Key Laboratory of Big Data Analysis and Application University of Science and Technology of China CAS Key Laboratory of Technology in Geo-Spatial Information Processing and Application System University of Science and Technology of China Tencent TaiQ Team

For a long time, object detectors have suffered from extreme imbalance between foregrounds and backgrounds. While several sampling/reweighting schemes have been explored to alleviate the imbalance, they are usually heuristic and demand laborious hyper-parameters tuning, which is hard to achieve the optimality. In this paper, we first reveal that such the imbalance could be addressed in a learning-based manner. Guided by this illuminating observation, we propose a novel Residual Objectness (ResObj) mechanism that addresses the imbalance by end-to-end optimization, while no further hand-crafted sampling/reweighting is required. Specifically, by applying multiple cascaded objectness-related modules with residual connections, we formulate an elegant consecutive refinement procedure for distinguishing the foregrounds from backgrounds, thereby progressively addressing the imbalance. Extensive experiments present the effectiveness of our method, as well as its compatibility and adaptivity for both region-based and one-stage detectors, namely, the RetinaNet-ResObj, YOLOv3-ResObj and FasterRCNN-ResObj achieve relative 3.6%, 3.9%, 3.2% Average Precision (AP) improvements compared with their vanilla models on COCO, respectively. Copyright © 2019, The Authors. All rights reserved.

关键词： Object detection

Faster distributed deep net training: Computation and communication decoupled stochastic gradient descent

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Shen, Shuheng Xu, Linli Liu, Jingchang Liang, Xianfeng Cheng, Yifei Anhui Province Key Laboratory of Big Data Analysis and Application School of Computer Science and Technology University of Science and Technology of China School of Data Science University of Science and Technology of China

With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization challenges. Nevertheless, although distributed stochastic gradient descent (SGD) algorithms can achieve a linear iteration speedup, they are limited significantly in practice by the communication cost, making it difficult to achieve a linear time speedup. In this paper, we propose a computation and communication decoupled stochastic gradient descent (CoCoDSGD) algorithm to run computation and communication in parallel to reduce the communication cost. We prove that CoCoD-SGD has a linear iteration speedup with respect to the total computation capability of the hardware resources. In addition, it has a lower communication complexity and better time speedup comparing with traditional distributed SGD algorithms. Experiments on deep neural network training demonstrate the significant improvements of CoCoD-SGD: when training ResNet18 and VGG16 with 16 Geforce GTX 1080Ti GPUs, CoCoD-SGD is up to 2-3x faster than traditional synchronous SGD. Copyright © 2019, The Authors. All rights reserved.

关键词： Program processors

QuesNet: A unified representation for heterogeneous test questions

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Yin, Yu Liu, Qi Huang, Zhenya Chen, Enhong Tong, Wei Wang, Shijin Su, Yu Anhui Province Key Laboratory of Big Data Analysis and Application School of Computer Science and Technology School of Data Science University of Science and Technology of China IFLYTEK Research IFLYTEK CO. LTD. State Key Laboratory of Cognitive Intelligence

Understanding learning materials (e.g. test questions) is a crucial issue in online learning systems, which can promote many applications in education domain. Unfortunately, many supervised approaches suffer from the problem of scarce human labeled data, whereas abundant unlabeled resources are highly underutilized. To alleviate this problem, an effective solution is to use pre-trained representations for question understanding. However, existing pretraining methods in NLP area are infeasible to learn test question representations due to several domain-specific characteristics in education. First, questions usually comprise of heterogeneous data including content text, images and side information. Second, there exists both basic linguistic information as well as domain logic and knowledge. To this end, in this paper, we propose a novel pre-training method, namely QuesNet, for comprehensively learning question representations. Specifically, we first design a unified framework to aggregate question information with its heterogeneous inputs into a comprehensive vector. Then we propose a two-level hierarchical pre-training algorithm to learn better understanding of test questions in an unsupervised way. Here, a novel holed language model objective is developed to extract low-level linguistic features, and a domain-oriented objective is proposed to learn high-level logic and knowledge. Moreover,we showthat Ques- Net has good capability of being fine-tuned in many question-based tasks. We conduct extensive experiments on large-scale real-world question data, where the experimental results clearly demonstrate the effectiveness of QuesNet for question understanding as well as its superior applicability. Copyright © 2019, The Authors. All rights reserved.

关键词： Learning systems

Illuminating Recommendation by Understanding the Explicit Item Relations

学校读者我要写书评

暂无评论

Journal of Computer Science & Technology 2018年第4期33卷 739-755页

作者： Qi Liu Hong-Ke Zhao Le Wu Zhi Li En-Hong Chen Anhui Province Key Laboratory of Big Data Analysis and Application University of Science and Technology of China Hefei 230027 China School of Computer and Information Hefei University of Technology Hefei 230009 China School of Software Engineering University of Science and Technology of China Hefei 230051 China

Recent years have witnessed the prevalence of recommender systems in various fields, which provide a personalized recommendation list for each user based on various kinds of information. For quite a long time, most researchers have been pursing recommendation performances with predefined metrics, e.g., accuracy. However, in real-world applications, users select items from a huge item list by considering their internal personalized demand and external constraints. Thus, we argue that explicitly modeling the complex relations among items under domain-specific applications is an indispensable part for enhancing the recommendations. Actually, in this area, researchers have done some work to understand the item relations gradually from ＂implicit＂ to ＂explicit＂ views when recommending. To this end, in this paper, we conduct a survey of these recent advances on recommender systems from the perspective of the explicit item relation understanding. We organize these relevant studies from three types of item relations, i.e., combination-effect relations, sequence-dependence relations, and external-constraint relations. Specifically, the combination-effect relation and the sequence-dependence relation based work models the intra-group intrinsic relations of items from the user demand perspective, and the external-constraint relation emphasizes the external requirements for items. After that, we also propose our opinions on the open issues along the line of understanding item relations and suggest some future research directions in recommendation area.

关键词： recommender system item relation recommendation interpretability

ICSTrace: A Malicious IP Traceback Model for Attacking data of Industrial Control System

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Xiao, Feng Xu, Qiang Anhui Province Key Laboratory of Big Data Analysis and Application School of Computer Science and Technology University of Science and Technology of China Electronic Engineering Institute of Hefei China

Considering the attacks against industrial control system are mostly organized and premeditated actions, IP traceback is significant for the security of industrial control system. Based on the infrastructure of the Internet, we have developed a novel malicious IP traceback model-ICSTrace, without deploying any new services. The model extracts the function codes and their parameters from the attack data according to the format of industrial control protocol, and employs a short sequence probability method to transform the function codes and their parameter into a vector, which characterizes the attack pattern of malicious IP addresses. Furthermore, a Partial Seeded K-Means algorithm is proposed for the pattern’s clustering, which helps in tracing the attacks back to an organization. ICSTrace is evaluated basing on the attack data captured by the large-scale deployed honeypots for industrial control system, and the results demonstrate that ICSTrace is effective on malicious IP traceback in industrial control system. Copyright © 2019, The Authors. All rights reserved.

关键词： K-means clustering

A New Free-Standing Aqueous Zinc-Ion Capacitor Based on MnO2–CNTs Cathode and MXene Anode

学校读者我要写书评

暂无评论

Nano-Micro Letters 2019年第4期11卷 227-238页

作者： Siliang Wang Qiang Wang Wei Zeng Min Wang Limin Ruan Yanan Ma Key Laboratory of Intelligent Computing and Signal Processing Ministry of EducationAnhui UniversityNo.3 Feixi RoadHefei 230039Anhui ProvincePeople’s Republic of China National Engineering Research Center for Agro-Ecological Big Data Analysis and Application School of Electronics and Information EngineeringAnhui UniversityNo.111 Jiulong RoadHefei 230601Anhui ProvincePeople’s Republic of China School of Sciences Hubei University of Automotive TechnologyNo.167 Checheng West RoadShiyan 442002Hubei ProvincePeople’s Republic of China

Restricted by their energy storage mechanism,current energy storage devices have certain drawbacks,such as low power density for batteries and low energy density for ***,the nearest ion capacitors,such as lithium-ion and sodium-ion capacitors containing battery-type and capacitor-type electrodes,may allow achieving both high energy and power *** the inspiration,a new zinc-ion capacitor(ZIC)has been designed and realized by assembling the free-standing manganese dioxide-carbon nanotubes(MnO2-CNTs)battery-type cathode and MXene(Ti3C2Tx)capacitortype anode in an aqueous *** ZIC can avoid the insecurity issues that frequently occurred in lithium-ion and sodium-ion capacitors in organic *** expected,the ZIC in an aqueous liquid electrolyte exhibits excellent electrochemical performance(based on the total weight of cathode and anode),such as a high specific capacitance of 115.1 F g?1(1 mV s?1),high energy density of 98.6 Wh kg?1(77.5 W kg?1),high power density of 2480.6 W kg?1(29.7 Wh kg?1),and high capacitance retention of^83.6%of its initial capacitance(15,000 cycles).Even in an aqueous gel electrolyte,the ZIC also exhibits excellent *** work provides an essential strategy for designing next-generation high-performance energy storage devices.

关键词： Energy storage Zinc-ion capacitor Battery-type and capacitor-type electrodes MXene Electrochemical performance

Alpha-Beta Sampling for Pairwise Ranking in One-Class Collaborative Filtering

学校读者我要写书评

暂无评论

Alpha-Beta Sampling for Pairwise Ranking in One-Class Collab...

IEEE International Conference on data Mining (ICDM)

作者： Mingyue Cheng Runlong Yu Qi Liu Vincent W. Zheng Hongke Zhao Hefu Zhang Enhong Chen Anhui Province Key Laboratory of Big Data Analysis and Application University of S&T of China Hefei China Tianjin University Tianjin China WeBank Shenzhen China

This paper introduces Alpha-Beta Sampling (ABS) strategy, which is particularly intended for the sampling problem of pairwise ranking in one-class collaborative filtering (PROCCF). Specifically, ABS strategy places more emphasis on such training examples, including positive item with a lower preference score and negative items with a higher preference score for each gradient step. Then, we provide the corresponding proofs for the ABS strategy from both gradient and ranking perspectives. First, we prove that sampled training examples by ABS strategy can update the model parameters with a large magnitude and analyze two instantiations by combining two specific pairwise algorithms. Second, it can be proved that ABS strategy is equivalent to optimizing for ranking-aware evaluation metrics like Normalized Discounted Cumulative Gain (NDCG). Furthermore, ABS strategy can be very general and applicable in a lot of pairwise structures of pairwise algorithms. Based on ABS strategy, we provide an effective sampling algorithm to dynamically draw items for each SGD update. Finally, we evaluate the ABS strategy by conducting sampling tasks in two representative pairwise algorithms. The experiment results show that the ABS strategy performs significantly better than the baseline strategies.

关键词：