检索结果-内蒙古大学图书馆

Compressed deep networks: Goodbye SVD, hello robust low-rank approximation

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Tukan, Murad Maalouf, Alaa Weksler, Matan Feldman, Dan Robotics and Big Data Lab Department of Computer Science University of Haifa Haifa Israel Samsung Research Israel Israel

A common technique for compressing a neural network is to compute the k-rank `2 approximation Ak,2 of the matrix A ∈ Rn×d that corresponds to a fully connected layer (or embedding layer). Here, d is the number of the neurons in the layer, n is the number in the next one, and Ak,2 can be stored in O((n + d)k) memory instead of O(nd). This l2-approximation minimizes the sum over every entry to the power of p = 2 in the matrix A − Ak,2, among every matrix Ak,2 ∈ Rn×d whose rank is k. While it can be computed efficiently via SVD, the l2-approximation is known to be very sensitive to outliers ("far-away" rows). Hence, machine learning uses e.g. Lasso Regression, `1-regularization, and l1-SVM that use the l1-norm. This paper suggests to replace the k-rank l2 approximation by lp, for p ∈ [1, 2]. We then provide practical and provable approximation algorithms to compute it for any p ≥ 1, based on modern techniques in computational geometry. Extensive experimental results on the GLUE benchmark for compressing BERT, DistilBERT, XLNet, and RoBERTa confirm this theoretical advantage. For example, our approach achieves 28% compression of RoBERTa’s embedding layer with only 0.63% additive drop in the accuracy (without fine-tuning) in average over all tasks in GLUE, compared to 11% drop using the existing l2-approximation. Open code is provided for reproducing and extending our results. Copyright © 2020, The Authors. All rights reserved.

关键词： Embeddings

BroadCAM: Outcome-agnostic Class Activation Mapping for Small-scale Weakly Supervised Applications

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Lin, Jiatai Han, Guoqiang Xu, Xuemiao Liang, Changhong Wong, Tien-Tsin Chen, C.L. Philip Liu, Zaiyi Han, Chu The School of Computer Science and Engineering South China University of Technology Guangdong Guangzhou510006 China Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application Guangzhou510080 China Southern Medical University Guangzhou510080 China School of Computer Science and Engineering South China University of Technology State Key Laboratory of Subtropical Building Science Ministry of Education Key Laboratory of Big Data and Intelligent Robot Guangdong Provincial Key Lab of Computational Intelligence and Cyberspace Information China Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong

Class activation mapping (CAM), a visualization technique for interpreting deep learning models, is now commonly used for weakly supervised semantic segmentation (WSSS) and object localization (WSOL). It is the weighted aggregation of the feature maps by activating the high class-relevance ones. Current CAM methods achieve it relying on the training outcomes, such as predicted scores (forward information), gradients (backward information), etc. However, when with small-scale data, unstable training may lead to less effective model outcomes and generate unreliable weights, finally resulting in incorrect activation and noisy CAM seeds. In this paper, we propose an outcome-agnostic CAM approach, called BroadCAM, for small-scale weakly supervised applications. Since broad learning system (BLS) is independent to the model learning, BroadCAM can avoid the weights being affected by the unreliable model outcomes when with small-scale data. By evaluating BroadCAM on VOC2012 (natural images) and BCSS-WSSS (medical images) for WSSS and OpenImages30k for WSOL, BroadCAM demonstrates superior performance than existing CAM methods with small-scale data (less than 5%) in different CNN architectures. It also achieves SOTA performance with large-scale training data. Extensive qualitative comparisons are conducted to demonstrate how BroadCAM activates the high class-relevance feature maps and generates reliable CAMs when with small-scale training data. © 2023, CC BY.

关键词： Learning systems

A systematic survey of PRMT interactomes reveals the key roles of arginine methylation in the global control of RNA splicing and translation

学校读者我要写书评

暂无评论

science Bulletin 2021年第13期66卷 1342-1357,M0004页

作者： Huan-Huan Wei Xiao-Juan Fan Yue Hu Xiao-Xu Tian Meng Guo Miao-Wei Mao Zhao-Yuan Fang Ping Wu Shuai-Xin Gao Chao Peng Yun Yang Zefeng Wang CAS Key Laboratory of Computational Biology Bio-Med Big Data CenterShanghai Institute of Nutrition and HealthCAS Center for Excellence in Molecular Cell ScienceUniversity of Chinese Academy of SciencesChinese Academy of SciencesShanghai 200031China National Facility for Protein Science in Shanghai Zhang-Jiang LabShanghai Advanced Research InstituteChinese Academy of SciencesShanghai 201210China Xijing Hospital of Digestive Diseases Fourth Military Medical UniversityXi’an 710000China

Thousands of proteins undergo arginine methylation,a widespread post-translational modification catalyzed by several protein arginine methyltransferases(PRMTs).However,global understanding of their biological functions is limited due to the lack of a complete picture of the catalytic network for each ***,we systematically identified interacting proteins for all human PRMTs and demonstrated their functional importance in mRNA splicing and *** demonstrated significant overlapping of interactomes of human PRMTs with the known methylarginine-containing *** PRMTs are functionally redundant with a high degree of overlap in their substrates and high similarities between their putative methylation ***,RNA-binding proteins involved in regulating RNA splicing and translation contain highly enriched arginine methylation ***,inhibition of PRMTs globally alternates alternative splicing(AS)and suppresses *** particular,ribosomal proteins are extensively modified with methylarginine,and mutations in their methylation sites suppress ribosome assembly,translation,and eventually cell ***,our study provides a global view of different PRMT networks and uncovers critical functions of arginine methylation in regulating mRNA splicing and translation.

关键词： Post-translational modification Protein arginine methyltransferase RNA-binding protein Ribosomal proteins Alternative splicing mRNA translation

A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Meng, Xiangxi Gu, Yuning Pan, Yongsheng Wang, Nizhuan Xue, Peng Lu, Mengkang He, Xuming Zhan, Yiqiang Shen, Dinggang School of Biomedical Engineering ShanghaiTech University Shanghai China School of Information Science and Technology ShanghaiTech University Shanghai China Shanghai United Imaging Intelligence Co. Ltd. Shanghai China National Engineering Lab. for Integrated Aero-Space-Ground-Ocean Big Data Application Technology School of Computer Science and Engineering Northwestern Polytechnical University Shaanxi Xi’an China

Multi-modal medical image completion has been extensively applied to alleviate the missing modality issue in a wealth of multi-modal diagnostic tasks. However, for most existing synthesis methods, their inferences of missing modalities can collapse into a deterministic mapping from the available ones, ignoring the uncertainties inherent in the cross-modal relationships. Here, we propose the Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM) to take advantage of Score-based Generative Model (SGM) in modeling and stochastically sampling a target probability distribution, and further extend SGM to cross-modal conditional synthesis for various missing-modality configurations in a unified framework. Specifically, UMM-CSGM employs a novel multi-in multi-out Conditional Score Network (mm-CSN) to learn a comprehensive set of cross-modal conditional distributions via conditional diffusion and reverse generation in the complete modality space. In this way, the generation process can be accurately conditioned by all available information, and can fit all possible configurations of missing modalities in a single network. Experiments on BraTS19 dataset show that the UMM-CSGM can more reliably synthesize the heterogeneous enhancement and irregular area in tumor-induced lesions for any missing modalities. Copyright © 2022, The Authors. All rights reserved.

关键词： Probability distributions

Divide and contrast: source-free domain adaptation via adaptive contrastive learning 22

学校读者我要写书评

暂无评论

Divide and contrast: source-free domain adaptation via adapt...

Proceedings of the 36th International Conference on Neural Information Processing Systems

作者： Ziyi Zhang Weikai Chen Hui Cheng Zhen Li Siyuan Li Liang Lin Guanbin Li National Key Laboratory of Novel Software Technology Nanjing University Nanjing China Tencent America School of Computer Science and Engineering Sun Yat-sen University Guangzhou China The Chinese University of Hong Kong Shenzhen China and Shenzhen Research Institute of Big Data Shenzhen China AI Lab School of Engineering Westlake University Hangzhou China

ISBN: (纸本)9781713871088

We investigate a practical domain adaptation task, called source-free unsupervised domain adaptation (SFUDA), where the source pretrained model is adapted to the target domain without access to the source data. Existing techniques mainly leverage self-supervised pseudo-labeling to achieve class-wise global alignment [1] or rely on local structure extraction that encourages the feature consistency among neighborhoods [2]. While impressive progress has been made, both lines of methods have their own drawbacks - the "global" approach is sensitive to noisy labels while the "local" counterpart suffers from the source bias. In this paper, we present Divide and Contrast (DaC), a new paradigm for SFUDA that strives to connect the good ends of both worlds while bypassing their limitations. Based on the prediction confidence of the source model, DaC divides the target data into source-like and target-specific samples, where either group of samples is treated with tailored goals under an adaptive contrastive learning framework. Specifically, the source-like samples are utilized for learning global class clustering thanks to their relatively clean labels. The more noisy target-specific data are harnessed at the instance level for learning the intrinsic local structures. We further align the sourcelike domain with the target-specific samples using a memory-based maximum mean discrepancy (MMD) loss to reduce the distribution mismatch. Extensive experiments on VisDA, Office-Home, and the more challenging DomainNet have verified the superior performance of DaC over current state-of-the-art approaches.

关键词：

Heterogeneous Graph Neural Network for Privacy-Preserving Recommendation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Wei, Yuecen Fu, Xingcheng Sun, Qingyun Peng, Hao Wu, Jia Wang, Jinyan Li, Xianxian Guangxi Key Lab of Multi-source Information Mining & Security Guangxi Normal University Guilin China School of Computer Science and Engineering Guangxi Normal University Guilin China Beijing Advanced Innovation Center for Big Data and Brain Computing Beihang University Beijing China School of Computer Science and Engineering Beihang University Beijing China School of Computing Macquarie University Sydney Australia

Social networks are considered to be heterogeneous graph neural networks (HGNNs) with deep learning technological advances. HGNNs, compared to homogeneous data, absorb various aspects of information about individuals in the training stage. That means more information has been covered in the learning result, especially sensitive information. However, the privacy-preserving methods on homogeneous graphs only preserve the same type of node attributes or relationships, which cannot effectively work on heterogeneous graphs due to the complexity. To address this issue, we propose a novel heterogeneous graph neural network privacy-preserving method based on a differential privacy mechanism named HeteDP, which provides a double guarantee on graph features and topology. In particular, we first define a new attack scheme to reveal privacy leakage in the heterogeneous graphs. Specifically, we design a two-stage pipeline framework, which includes the privacy-preserving feature encoder and the heterogeneous link reconstructor with gradients perturbation based on differential privacy to tolerate data diversity and against the attack. To better control the noise and promote model performance, we utilize a bi-level optimization pattern to allocate a suitable privacy budget for the above two modules. Our experiments on four public benchmarks show that the HeteDP method is equipped to resist heterogeneous graph privacy leakage with admirable model generalization. Copyright © 2022, The Authors. All rights reserved.

关键词： Budget control

Superpage-Friendly Page Table Design for Hybrid Memory Systems 6th

学校读者我要写书评

暂无评论

Superpage-Friendly Page Table Design for Hybrid Memory Syste...

6th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2020

作者： Wang, Xiaoyuan Liu, Haikun Liao, Xiaofei Jin, Hai National Engineering Research Center for Big Data Technology and System Huazhong University of Science and Technology Wuhan430074 China Service Computing Technology and System Lab Huazhong University of Science and Technology Wuhan430074 China Cluster and Grid Computing Lab Huazhong University of Science and Technology Wuhan430074 China School of Computer Science and Technology Huazhong University of Science and Technology Wuhan430074 China

ISBN: (纸本)9789811579806

Page migration has long been adopted in hybrid memory systems comprising dynamic random access memory (DRAM) and non-volatile memories (NVMs), to improve the system performance and energy efficiency. However, page migration introduces some side effects, such as more translation lookaside buffer (TLB) misses, breaking memory contiguity, and extra memory accesses due to page table updating. In this paper, we propose superpage-friendly page table called SuperPT to reduce the performance overhead of serving TLB misses. By leveraging a virtual hashed page table and a hybrid DRAM allocator, SuperPT performs address translations in a flexible and efficient way while still remaining the contiguity within the migrated pages. © 2020, Springer Nature Singapore Pte Ltd.

关键词： Energy efficiency

Heterogeneous Graph Neural Network for Privacy-Preserving Recommendation

学校读者我要写书评

暂无评论

Heterogeneous Graph Neural Network for Privacy-Preserving Re...

IEEE International Conference on data Mining (ICDM)

作者： Yuecen Wei Xingcheng Fu Qingyun Sun Hao Peng Jia Wu Jinyan Wang Xianxian Li Guangxi Key Lab of Multi-source Information Mining & Security Guangxi Normal University Guilin China School of Computer Science and Engineering Guangxi Normal University Guilin China Beijing Advanced Innovation Center for Big Data and Brain Computing Beihang University Beijing China School of Computer Science and Engineering Beihang University Beijing China School of Computing Macquarie University Sydney Australia

关键词： Training Privacy Differential privacy Social networking (online) Graph neural networks Topology Resource management

GrabDAE: An Innovative Framework for Unsupervised Domain Adaptation Utilizing Grab-Mask and Denoise Auto-Encoder

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Chen, Junzhou Wen, Xuan Zhang, Ronghui Ren, Bingtao Wu, Di Xu, Zhigang Wang, Danwei The Guangdong Provincial Key Laboratory of Intelligent Transport System School of Intelligent Systems Engineering Sun Yat-sen University Guangzhou510275 China The School of Transportation Science and Engineering Beihang University State Key Lab of Intelligent Transportation System Beijing100191 China The School of Computer Science and Engineering Sun Yat-sen University Guangzhou510006 China The Guangdong Key Laboratory of Big Data Analysis and Processing Guangdong510006 China The School of Information Engineering Chang’an University Shaanxi Xi’an710064 China The School of Electrical and Electronic Engineering Nanyang Technological University Singapore639798 Singapore

Unsupervised Domain Adaptation (UDA) aims to adapt a model trained on a labeled source domain to an unlabeled target domain by addressing the domain shift. Existing Unsupervised Domain Adaptation (UDA) methods often fall short in fully leveraging contextual information from the target domain, leading to suboptimal decision boundary separation during source and target domain alignment. To address this, we introduce GrabDAE, an innovative UDA framework designed to tackle domain shift in visual classification tasks. GrabDAE incorporates two key innovations: the Grab-Mask module, which blurs background information in target domain images, enabling the model to focus on essential, domain-relevant features through contrastive learning;and the Denoising Auto-Encoder (DAE), which enhances feature alignment by reconstructing features and filtering noise, ensuring a more robust adaptation to the target domain. These components empower GrabDAE to effectively handle unlabeled target domain data, significantly improving both classification accuracy and robustness. Extensive experiments on benchmark datasets, including VisDA-2017, OfficeHome, and Office31, demonstrate that GrabDAE consistently surpasses state-of-the-art UDA methods, setting new performance benchmarks. By tackling UDA’s critical challenges with its novel feature masking and denoising approach, GrabDAE offers both significant theoretical and practical advancements in domain adaptation. © 2024, CC BY-NC-SA.

关键词：