检索结果-内蒙古大学图书馆

Variational Distillation for Multi-View Learning

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024年第7期46卷 4551-4566页

作者： Tian, Xudong Zhang, Zhizhong Wang, Cong Zhang, Wensheng Qu, Yanyun Ma, Lizhuang Wu, Zongze Xie, Yuan Tao, Dacheng East China Normal Univ Sch Comp Sci & Technol Shanghai 200062 Peoples R China Huawei Technol Distributed & Parallel Software Lab Labs 2012 Hangzhou 518129 Peoples R China Chinese Acad Sci Inst Automat Beijing 100190 Peoples R China Xiamen Univ Sch Informat Sci & Technol Xiamen 361005 Fujian Peoples R China East China Normal Univ Sch Comp Sci & Software Engn Shanghai 200050 Peoples R China Shanghai Jiao Tong Univ Sch Elect Informat & Elect Engn Shanghai 200240 Peoples R China Shenzhen Univ Coll Mech & Control Engn Shenzhen 518060 Peoples R China Normal Univ Chongqing Inst East China Shanghai 200062 Peoples R China JD Exploer Acad Beijing Peoples R China Univ Sydney Camperdown NSW 2050 Australia

Information Bottleneck (IB) provides an information-theoretic principle for multi-view learning by revealing the various components contained in each viewpoint. This highlights the necessity to capture their distinct roles to achieve view-invariance and predictive representations but remains under-explored due to the technical intractability of modeling and organizing innumerable mutual information (MI) terms. Recent studies show that sufficiency and consistency play such key roles in multi-view representation learning, and could be preserved via a variational distillation framework. But when it generalizes to arbitrary viewpoints, such strategy fails as the mutual information terms of consistency become complicated. This paper presents Multi-View Variational Distillation ((MVD)-D-2), tackling the above limitations for generalized multi-view learning. Uniquely, (MVD)-D-2 can recognize useful consistent information and prioritize diverse components by their generalization ability. This guides an analytical and scalable solution to achieving both sufficiency and consistency. Additionally, by rigorously reformulating the IB objective, (MVD)-D-2 tackles the difficulties in MI optimization and fully realizes the theoretical advantages of the information bottleneck principle. We extensively evaluate our model on diverse tasks to verify its effectiveness, where the considerable gains provide key insights into achieving generalized multi-view representations under a rigorous information-theoretic principle.

关键词： Mutual information Task analysis Representation learning Predictive models Optimization Visualization Pattern analysis Multi-view learning information bottleneck mutual information variational inference knowledge distillation

来源：评论

学校读者我要写书评

暂无评论

Pre-training for ad-hoc retrieval: Hyperlink is also you need

arXiv

引用

arXiv 2021年

作者： Ma, Zhengyi Dou, Zhicheng Xu, Wei Zhang, Xinyu Jiang, Hao Cao, Zhao Wen, Ji-Rong Gaoling School of Artificial Intelligence Renmin University of China Beijing China Distributed and Parallel Software Lab Huawei Beijing Key Laboratory of Big Data Management and Analysis Methods Beijing China Key Laboratory of Data Engineering and Knowledge Engineering MOE Beijing China

Designing pre-training objectives that more closely resemble the downstream tasks for pre-trained language models can lead to better performance at the fine-tuning stage, especially in the ad-hoc retrieval area. Existing pre-training approaches tailored for IR tried to incorporate weak supervised signals, such as query-likelihood based sampling, to construct pseudo query-document pairs from the raw textual corpus. However, these signals rely heavily on the sampling method. For example, the query likelihood model may lead to much noise in the constructed pre-training data. In this paper, we propose to leverage the large-scale hyperlinks and anchor texts to pre-train the language model for ad-hoc retrieval. Since the anchor texts are created by webmasters and can usually summarize the target document, it can help to build more accurate and reliable pre-training samples than a specific algorithm. Considering different views of the downstream ad-hoc retrieval, we devise four pre-training tasks based on the hyperlinks. We then pre-train the Transformer model to predict the pair-wise preference, jointly with the Masked Language Model objective. Experimental results on two large-scale ad-hoc retrieval datasets show the significant improvement of our model compared with the existing methods. Copyright © 2021, The Authors. All rights reserved.

关键词： Hypertext systems

来源：评论

学校读者我要写书评

暂无评论

PPSS: A Privacy-Preserving Supervision Scheme Based on Blockchain

PPSS: A Privacy-Preserving Supervision Scheme Based on Block...

引用

IEEE International Conference on Big Data and Cloud Computing (BdCloud)

作者： Ming Chen Peichang Shi Xiang Fu Feng Jiang Fei Gao Penghui Ma Jinzhu Kong National Key Laboratory of Parallel and Distributed Processing College of Computer Science National University of Defense Technology Changsha China Key Laboratory of Software Engineering for Complex Systems College of Computer Science National University of Defense Technology Changsha China Kylin Software Co.LTD Changsha China Zhejiang Lab Hangzhou China

In JointCloud Computing, multi-party participation introduces complexity and uncertainty. For all participants in JointCloud Computing, both continuous supervision and necessary privacy protection are required. Traditional supervision methods usually adopt the centralized information interaction mode, which has such defects as collusion of interests, single point of failure, privacy disclosure, etc. Building a decentralized supervision mechanism has become a new research direction. In this paper, we propose PPSS, a privacy-preserving supervision scheme based on blockchain, which decentralizes the supervision of the participants in JointCloud Computing, and combines the “double encryptions” and “threshold encryption” technologies to provide privacy protection. While making full use of the decentralization of the blockchain, a committee is established to carry out the analysis and decision-making tasks in terms of supervision and privacy protection. Experimental results indicate that PPSS can balance performance and security by reasonably configuring the committee.

关键词： Privacy Uncertainty Decision making Buildings Reliability engineering Blockchains Complexity theory

来源：评论

学校读者我要写书评

暂无评论

distributed and parallel Sparse Computing for Very Large Graph Neural Networks

Distributed and Parallel Sparse Computing for Very Large Gra...

引用

IEEE International Conference on Big Data

作者： Quentin R. Petit Chong Li Nahid Emad Huawei Paris Research Center & Université Paris-Saclay Boulogne-Billancourt France Distributed and Parallel Software Lab Huawei Paris Research Center Boulogne-Billancourt France Maison de la Simulation & LI-PaRAD Université Versailles Saint-Quentin Saclay France

ISBN: (纸本)9781665480468

Deep learning (DL) requires high-performance processing on big data. Graph Neural Networks, a challenging topic in DL using linear algebra methods, need algorithmic solutions to efficiently assign and process graph data on modern distributed and parallel machines, which are considered with mixed arithmetic and various types of tensor/matrix accelerators. Determining compression techniques for the graph’s sparse data structures is one of the key *** first objective is to design and implement a reusable parallel numerical library to resolve large neural network graphs. Our design strategy is drawn on a component-based approach and targets maximum code reuse in various parallel contexts while allowing for performance optimization. The solution could be later integrated into a DL framework like MindSpore.

关键词： Deep learning distributed databases Linear algebra Big Data parallel machines Graph neural networks Libraries

来源：评论

学校读者我要写书评

暂无评论

An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models

arXiv

引用

arXiv 2024年

作者： Liu, Qi Guo, Gang Mao, Jiaxin Dou, Zhicheng Wen, Ji-Rong Jiang, Hao Zhang, Xinyu Cao, Zhao Being Key Laboratory of Data Management and Analysis Methods Gaoling School of Artificial Intelligence Renmin University of China No.59 Zhongguancun Street Beijing100872 China Distributed and Parallel Software Lab Huawei Zhejiang No. 360 Jiangshu Road Hangzhou310056 China

With the development of pre-trained language models, the dense retrieval models have become promising alternatives to the traditional retrieval models that rely on exact match and sparse bag-of-words representations. Different from most dense retrieval models using a bi-encoder to encode each query or document into a dense vector, the recently proposed late-interaction multi-vector models (i.e., ColBERT and COIL) achieve state-of-the-art retrieval effectiveness by using all token embeddings to represent documents and queries and modeling their relevance with a sum-of-max operation. However, these fine-grained representations may cause unacceptable storage overhead for practical search systems. In this study, we systematically analyze the matching mechanism of these late-interaction models and show that the sum-of-max operation heavily relies on the co-occurrence signals and some important words in the document. Based on these findings, we then propose several simple document pruning methods to reduce the storage overhead and compare the effectiveness of different pruning methods on different late-interaction models. We also leverage query pruning methods to further reduce the retrieval latency. We conduct extensive experiments on both in-domain and out-domain datasets and show that some of the used pruning methods can significantly improve the efficiency of these late-interaction models without substantially hurting their retrieval effectiveness. © 2024, CC BY.

关键词： Efficiency

来源：评论

学校读者我要写书评

暂无评论

Improving knowledge distillation via an expressive teacher

引用

KNOWLEDGE-BASED SYSTEMS 2021年 218卷 106837-106837页

作者： Tan, Chao Liu, Jie Zhang, Xiang Natl Univ Def Technol Sci & Parallel & Distributed Proc Lab Changsha 410073 Peoples R China Natl Univ Def Technol Lab Software Engn Complex Syst Changsha 410073 Peoples R China Natl Univ Def Technol Inst Quantum Informat Changsha 410073 Peoples R China Natl Univ Def Technol State Key Lab High Performance Comp Changsha 410073 Peoples R China

Knowledge distillation (KD) is a widely used network compression technique for seeking a light student network with similar behaviors to its heavy teacher network. Previous studies mainly focus on training the student to mimic representation space of the teacher. However, how to be a good teacher is rarely explored. We find that if a teacher has weak ability to capture the knowledge underlying the true data in the real world, the student cannot even learn knowledge from its teacher. Inspired by that, we propose an inter-class correlation regularization to train teacher to capture a more explicit correlation among classes. Besides, we enforce student to mimic inter-class correlation of its teacher. Extensive experiments of image classification task have been conducted on four public benchmarks. For example, when the teacher and student networks are ShuffleNetV2-1.0 and ShuffleNetV2-0.5, our proposed method achieves 42.63% top-1 error rate for Tiny ImageNet. (C) 2021 Elsevier B.V. All rights reserved.

关键词： Neural network compression Knowledge distillation Knowledge transfer

来源：评论

学校读者我要写书评

暂无评论

An Efficient Scheme for Coupling OpenMC and FLUENT with Adaptive Load Balancing

引用

SCIENCE AND TECHNOLOGY OF NUCLEAR INSTALLATIONS 2021年第1期2021卷

作者： Zhang, Qingyang Peng, Tianji Zhang, Guangchun Liu, Jie Guo, Xiaowei Gong, Chunye Yang, Bo Fan, Xukai Natl Univ Def Technol Sci & Technol Parallel & Distributed Proc Lab Changsha 410073 Peoples R China Natl Univ Def Technol Lab Software Engn Complex Syst Changsha 410073 Peoples R China Chinese Acad Sci Inst Modern Phys Lanzhou 730030 Peoples R China Univ Chinese Acad Sci Sch Nucl Sci & Technol Beijing 100049 Peoples R China

This paper develops a multi-physics interface code MC-FLUENT to couple the Monte Carlo code OpenMC with the commercial computational fluid dynamics code ANSYS FLUENT. The implementations and parallel performances of block Gauss-Seidel-type and block Jacobi-type Picard iterative algorithms have been investigated. In addition, this paper introduces two adaptive load-balancing algorithms into the neutronics and thermal-hydraulics coupled simulation to reduce the time cost of computation. Considering that the different scalability of OpenMC and FLUENT limits the performance of block Gauss-Seidel algorithm, an adaptive load-balancing algorithm that can increase the number of nodes dynamically is proposed to improve its efficiency. Moreover, with the natural parallelism of block Jacobi algorithm, another adaptive load-balancing algorithm is proposed to improve its performance. A 3 x 3 PWR fuel pin model and a 1000 MWt ABR metallic benchmark core were used to compare the performances of the two algorithms and verify the effectiveness of the two adaptive load-balancing algorithms. The results show that the adaptive load-balancing algorithms proposed in this paper can greatly improve the computing efficiency of block Jacobi algorithm and improve the performance of block Gauss-Seidel algorithm when the number of nodes is large. In addition, the adaptive load-balancing algorithms are especially effective when a case demands different computational power of OpenMC and FLUENT.

关键词： Iterative methods

来源：评论

学校读者我要写书评

暂无评论

Online intervention siamese tracking

引用

INFORMATION SCIENCES 2023年第1期637卷

作者： Cai, Huayue Lan, Long Zhang, Jing Zhang, Xiang Xiao, Changcheng Luo, Zhigang Natl Univ Def Technol Inst Quantum Changsha Peoples R China Natl Univ Def Technol State Key Lab High Performance Comp Changsha Peoples R China Univ Sydney Sch Comp Sci Sydney Australia Natl Univ Def Technol Lab Digitizing Software Frontier Equipment Changsha Peoples R China Natl Univ Def Technol Coll Comp Sci & Technol Parallel & Distributed Lab Changsha Peoples R China

Online target update excels in helping a visual tracking algorithm adapt to variations in target appearance during inference, and thus is preferred by recent advances. However, this technique typically operates in a heuristic manner by updating numerous network parameters with large amounts of online-collected data. This poses a distinct challenge: can the online target update be effectively executed without using the aforementioned approach? To this end, we propose a novel target feature update scheme to reduce the need for tedious data collection and computation-intensive parameter updates. This scheme operates on the principle of causal intervention and is just as effective as default parameter updates in visual tracking. Besides, we explore a novel video-specific target label to capture the context of a specific target in video frames for feature discrimination. This makes target features better fit for appearance changes. Such schemes together with the off-shelf pre-trained classification backbone form a novel online intervention siamese tracker (OIS). When equipped with an unsupervised pre-trained backbone, OIS outperforms current state-of-the-art unsupervised trackers on the OTB and VOT. When exploiting a supervised trained backbone, it competes with typical supervised trackers trained on massive offline training and online tracking data.

关键词： Online siamese tracking Causal intervention Training-data free

来源：评论

学校读者我要写书评

暂无评论

Enhancing Graph Convolutional Networks by Topology Sampling

Enhancing Graph Convolutional Networks by Topology Sampling

引用

IEEE International Conference on Big Data

作者： Quentin R. Petit Chong Li Serge G. Petiton Kelun Chai Nahid Emad Huawei Paris Research Center & Université Paris-Saclay Boulogne-Billancourt France Distributed and Parallel Software Lab Huawei Paris Research Center Boulogne-Billancourt France Université de Lille Lille France Maison de la Simulation & LI-PaRAD Université Versailles Saint-Quentin Saclay France

ISBN: (纸本)9781665480468

Graph Neural Networks (GNNs) play a very important role today. It does analyze not only the graph data itself, but also the data connectivity of the graph. The quality of a GNN is thus altered by the result of extracted graph structure information. The extraction could be enhanced by GNN model design or directly from the training dataset with a GNN-decoupled method. In this paper, we propose RankedDrop, a new sampling method to improve the extraction of graph structure information. This approach is based on droppingout technique, and it adopts a spatial-aware selection of edges to drop. It takes into account structure information of the graph to control the dropping-out, and its random selection of edges to be dropped is under the control of a probability generated with respect to graph’s topological importance. Our experiments point out that RankedDrop provides high-quality and robust training results compared to the leading solutions. Furthermore, RankedDrop could be a framework plugin and combined with GNN model improvements to maximize GNN quality. Furthermore, RankedDrop could be a plugin for AI frameworks like MindSpore and combined with GNN model improvements to maximize GNN quality.

关键词： Training Network topology Big Data Sampling methods Graph neural networks Topology Data mining

来源：评论

学校读者我要写书评

暂无评论

Discriminative object tracking by domain contrast

引用

COMPUTER VISION AND IMAGE UNDERSTANDING 2024年 239卷

作者： Cai, Huayue Zhang, Xiang Lan, Long Xiao, Changcheng Xu, Chuanfu Liu, Jie Luo, Zhigang Natl Univ Def Technol Inst Quantum Changsha Peoples R China Natl Univ Def Technol State Key Lab High Performance Comp Changsha Peoples R China Natl Univ Def Technol Coll Comp Sci & Technol Parallel & Distributed Lab Changsha Peoples R China Natl Univ Def Technol Coll Comp Lab Software Engn Complex Syst Changsha Peoples R China

Multi-domain tracking method as a novel learning paradigm improves object tracking by sharing domain information with a common backbone whilst learning private information with domain-specific layers. In that context, each individual video sequence as a specific domain serves for a domain-specific layer. In this paper, we observe an intriguing finding that target features from different domains are highly confused with each other, thus having weak discriminative ability, for lack of domain interaction. To this end, we propose a simple yet effective domain interaction training paradigm called domain contrast to boost discriminative object features by effectively using amounts of instances from all the domains in two novel aspects: (1) a light-weight memory-saving training algorithm is proposed to solve the "out-of-the-memory"problem, which paves the way to couple with previous multi-domain trackers, and (2) a composite class-balanced loss is explored to tackle a more practical imbalanced problem, which not only involves the usual class imbalance problem but also accounts for the case of the totally mere negative instances. Experiments on multiple popular tracking benchmark datasets show that our mechanism consistently achieves the tracking performance gain of both base multi-domain tracker and its real-time variant thereof, without any other changes made on the original network.

关键词： Multi-domain tracking Domain interaction Discriminative features Class-balanced loss

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：