检索结果-内蒙古大学图书馆

ReCoSa: Detecting the relevant contexts with self-attention for multi-turn dialogue generation

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Zhang, Hainan Y., Lan L., Pang J., Guo X., Cheng CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing China

In multi-turn dialogue generation, response is usually related with only a few contexts. Therefore, an ideal model should be able to detect these relevant contexts and produce a suitable response accordingly. However, the widely used hierarchical recurrent encoder-decoder models just treat all the contexts indiscriminately, which may hurt the following response generation process. Some researchers try to use the cosine similarity or the traditional attention mechanism to find the relevant contexts, but they suffer from either insufficient relevance assumption or position bias problem. In this paper, we propose a new model, named ReCoSa, to tackle this problem. Firstly, a word level LSTM encoder is conducted to obtain the initial representation of each context. Then, the self-attention mechanism is utilized to update both the context and masked response representation. Finally, the attention weights between each context and response representations are computed and used in the further decoding process. Experimental results on both Chinese customer services dataset and English Ubuntu dialogue dataset show that ReCoSa significantly outperforms baseline models, in terms of both metric-based and human evaluations. Further analysis on attention shows that the detected relevant contexts by ReCoSa are highly coherent with human's understanding, validating the correctness and interpretability of ReCoSa. Copyright © 2019, The Authors. All rights reserved.

关键词： Signal encoding

GERE: Generative Evidence Retrieval for Fact Verification

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Chen, Jiangui Zhang, Ruqing Guo, Jiafeng Fan, Yixing Cheng, Xueqi CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing China

Fact verification (FV) is a challenging task which aims to verify a claim using multiple evidential sentences from trustworthy corpora, e.g., Wikipedia. Most existing approaches follow a three-step pipeline framework, including document retrieval, sentence retrieval and claim verification. High-quality evidences provided by the first two steps are the foundation of the effective reasoning in the last step. Despite being important, high-quality evidences are rarely studied by existing works for FV, which often adopt the off-the-shelf models to retrieve relevant documents and sentences in an "index-retrieve-then-rank" fashion. This classical approach has clear drawbacks as follows: i) a large document index as well as a complicated search process is required, leading to considerable memory and computational overhead;ii) independent scoring paradigms fail to capture the interactions among documents and sentences in ranking;iii) a fixed number of sentences are selected to form the final evidence set. In this work, we propose GERE, the first system that retrieves evidences in a generative fashion, i.e., generating the document titles as well as evidence sentence identifiers. This enables us to mitigate the aforementioned technical issues since: i) the memory and computational cost is greatly reduced because the document index is eliminated and the heavy ranking process is replaced by a light generative process;ii) the dependency between documents and that between sentences could be captured via sequential generation process;iii) the generative formulation allows us to dynamically select a precise set of relevant evidences for each claim. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines, with both time-efficiency and memory-efficiency. Copyright © 2022, The Authors. All rights reserved.

关键词： Efficiency

On layer normalization in the transformer architecture 37

学校读者我要写书评

暂无评论

On layer normalization in the transformer architecture

37th International Conference on Machine Learning, ICML 2020

作者： Xiong, Ruibin Yang, Yunchang He, Di Zheng, Kai Zheng, Shuxin Xing, Chen Zhang, Huishuai Lan, Yanyan Wang, Liwei Liu, Tie-Yan CAS Key Laboratory of Network Data Science and Technology Institute of Computing Technolog Chinese Academy of Sciences China University of Chinese Academy of Sciences China Center for Data Science Peking University Beijing Institute of Big Data Research China Key Laboratory of Machine Perception MOE School of EECS Peking University China Microsoft Research College of Computer Science Nankai University China

ISBN: (纸本)9781713821120

The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyperparameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications. Copyright 2020 by the author(s).

关键词： Mean field theory

Influence maximization with ε-almost submodular threshold functions 17

学校读者我要写书评

暂无评论

Influence maximization with ε-almost submodular threshold f...

Proceedings of the 31st International Conference on Neural Information Processing Systems

作者： Qiang Li Wei Chen Xiaoming Sun Jialin Zhang CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences and University of Chinese Academy of Sciences Microsoft Research

ISBN: (纸本)9781510860964

Influence maximization is the problem of selecting k nodes in a social network to maximize their influence spread. The problem has been extensively studied but most works focus on the submodular influence diffusion models. In this paper, motivated by empirical evidences, we explore influence maximization in the non-submodular regime. In particular, we study the general threshold model in which a fraction of nodes have non-submodular threshold functions, but their threshold functions are closely upper- and lower-bounded by some submodular functions (we call them ε-almost submodular). We first show a strong hardness result: there is no l/nγ/c approximation for influence maximization (unless P = NP) for all networks with up to nγ ε-almost submodular nodes, where γ is in (0,1) and c is a parameter depending on ε. This indicates that influence maximization is still hard to approximate even though threshold functions are close to submodular. We then provide (l - ε)ℓ(1 - 1/ε) approximation algorithms when the number of ε-almost submodular nodes is ℓ. Finally, we conduct experiments on a number of real-world datasets, and the results demonstrate that our approximation algorithms outperform other baseline algorithms.

关键词：

Blind separation of noncooperative paired carrier multiple access signals based on improved quantum-inspired evolutionary algorithm and receding horizon optimization

学校读者我要写书评

暂无评论

Blind separation of noncooperative paired carrier multiple a...

作者： Deng, Qi Zhang, Shanshan Chen, Gang Lu, Huaxiang Lab of Artificial Networks Institute of Semiconductors CAS Beijing China University of Chinese Academy of Sciences Beijing China CAS Center for Excellence in Brain Science and Intelligence Technology Chinese Academy of Sciences Beijing China Beijing Key Laboratory of Semiconductor Neural Network Intelligent Sensing and Computing Technology Beijing China

The single-channel blind source separation of paired carrier multiple access (PCMA) signal is a key technology in satellite communications. Due to the high-order complexity of existing separation methods and the uncertainty of channel parameters, the processing of noncooperative PCMA signals with long memory remains a great challenge. In this article, the blind separation was solved as the joint channel estimation and sequence detection, and a novel blind separation algorithm based on the improved quantum-inspired evolutionary algorithm (IQEA) and receding horizon optimization was proposed. Considering the practical scenarios, a mixed PCMA signal model with noninteger period sampling was presented in advance. Combined with the proposed signal model, the IQEA was applied to recover the uplink signals from the mixed PCMA signals by its iterative estimation. Meanwhile, to reduce the computational complexity caused by long memory of PCMA signals, the receding horizon optimization was introduced to process the whole signal sequence as a dynamic programming problem. The simulation results verified that the proposed RH-IQEA outperforms existing state-of-the-art separation algorithms in the blind separation of noncooperative PCMA signals, with higher accuracy, lower-order complexity and stronger robustness to parameter estimation errors. © 2021 John Wiley & Sons, Ltd.

关键词： Iterative methods

Beyond low-frequency information in graph convolutional networks

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Bo, Deyu Wang, Xiao Shi, Chuan Shen, Huawei Beijing University of Posts and Telecommunications China CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences Beijing China

Graph neural networks (GNNs) have been proven to be effective in various network-related tasks. Most existing GNNs usually exploit the low-frequency signals of node features, which gives rise to one fundamental question: is the low-frequency information all we need in the real world applications? In this paper, we first present an experimental investigation assessing the roles of low-frequency and high-frequency signals, where the results clearly show that exploring low-frequency signal only is distant from learning an effective node representation in different scenarios. How can we adaptively learn more information beyond low-frequency information in GNNs? A well-informed answer can help GNNs enhance the adaptability. We tackle this challenge and propose a novel Frequency Adaptation Graph Convolutional networks (FAGCN) with a self-gating mechanism, which can adaptively integrate different signals in the process of message passing. For a deeper understanding, we theoretically analyze the roles of low-frequency signals and high-frequency signals on learning node representations, which further explains why FAGCN can perform well on different types of networks. Extensive experiments on six real-world networks validate that FAGCN not only alleviates the over-smoothing problem, but also has advantages over the state-of-the-arts. Copyright © 2021, The Authors. All rights reserved.

关键词： Graph neural networks

EagleMine: Vision-guided mining in large graphs

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Feng, Wenjie Liu, Shenghua Faloutsos, Christos Hooi, Bryan Shen, Huawei Cheng, Xueqi Cas Key Laboratory of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences School of Computer Science Carnegie Mellon University

Given a graph with millions of nodes, what patterns exist in the distributions of node characteristics, and how can we detect them and separate anomalous nodes in a way similar to human vision? In this paper, we propose a vision-guided algorithm, EagleMine, to summarize micro-cluster patterns in two-dimensional histogram plots constructed from node features in a large graph. EagleMine utilizes a water-level tree to capture cluster structures according to vision-based intuition at multi-resolutions. EagleMine traverses the waterlevel tree from the root and adopts statistical hypothesis tests to determine the optimal clusters that should be fitted along the path, and summarizes each cluster with a truncated Gaussian distribution. Experiments on real data show that our method can find truncated and overlapped elliptical clusters, even when some baseline methods split one visual cluster into pieces with Gaussian spheres. To identify potentially anomalous micro-clusters, EagleMine also a designates score to measure the suspiciousness of outlier groups (i.e. node clusters) and outlier nodes, detecting bots and anomalous users with high accuracy in the real Microblog data. Copyright © 2017, The Authors. All rights reserved.

关键词： Statistics

A Novel Multi-Edge Cooperative Offloading Genetic Algorithm towards Environmental Monitoring System 5

学校读者我要写书评

暂无评论

A Novel Multi-Edge Cooperative Offloading Genetic Algorithm ...

5th International Conference on Telecommunications and Communication Engineering, ICTCE 2022

作者： Fang, Jie Li, Wenzao Wang, Yue Sun, Xiulan College of Communication Engineering Chengdu University of Information Technology China Network and Data Security Key Lab of Sichuan Pro. University of Electronic Science and Technology of China China Institute of Data Application Research Educational Informalization and Big Data Center China

ISBN: (纸本)9781450397797

In environmental monitor architecture, the usual way of processing data is centralized. On the one hand, this leads to a long time of data processing, and with the increasing number of monitoring sensors, the pressure of data processing in data centers increases sharply. The edge computing provides a feasible solution to relieve the processing pressure and decrease the delay of data processing in data center. However, in the environmental monitor system, the edge servers are often load unbalance because of the nonuniform distribution of sensors. While, it is usually ignored in recently researches. In this paper, we proposed a cooperative offloading strategy (MECOGA) based on genetic algorithm (GA). It aims to ease the data processing pressure and reduce the data processing time in the data center. In addition, the load balance of edge servers is also one of our optimization goals. In this paper, we aim at decreasing task processing delay and improving the system load balancing. Contrast to other algorithms, our MECOGA decreases the delay near 20% before collaborative loading. And the balance variance is 15% smaller than the Balance strategy. © 2022 Association for computing Machinery.

关键词： Genetic algorithms

DFS-NER: Description Enhanced Few-shot NER via Prompt Learning and Meta-Learning

学校读者我要写书评

暂无评论

DFS-NER: Description Enhanced Few-shot NER via Prompt Learni...

2022 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent technology (WI-IAT)

作者： Huinan Huang Yuming Feng Xiaolong Jin Saiping Guan Jiafeng Guo School of Computer Science and Technology University of Chinese Academy of Sciences CAS Key Laboratory of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences

Named Entity Recognition (NER) is a very common task in many social good related domains. Recently, deep learning based NER has gradually matured, but still faces the scarcity problem of labeled data in specific domains. Therefore, researchers focus on few-shot NER to reduce the model’s data dependence and enhance the transferability of the model. However, existing works usually cannot adapt to new entity types and are prone to the so-called negative transfer problem. Therefore, in this paper we propose a type- Description-enhanced Few Shot NER model, called DFS-NER, which effectively integrates the prompt learning paradigm and the meta-learning framework. DFS-NER performs well under frozen pre-training model parameters through continuous templates. It realizes efficient source domain training and target domain parameter fine-tuning through the metalearning framework. We enhance the robustness of entity- type prototype representations by introducing word-word- level and word-type-level contrastive learning objectives and capsule networks as the induction module. Simultaneously, based on discrete prompt learning, a masked-language model learning objective guided by type description is proposed, which can well absorb the semantic information of entity types. Experiments on commonly used datasets, including, SNIPS, Few-NERD, and MIT Movie show that DFS-NER basically surpasses baseline models and achieves the state-of-the-art performance.

关键词： Deep learning Training Adaptation models Analytical models Semantics Prototypes Robustness