检索结果-内蒙古大学图书馆

SP3: Enhancing Structured Pruning via PCA Projection

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Hu, Yuxuan Zhang, Jing Zhao, Zhe Zhao, Chen Chen, Xiaodong Li, Cuiping Chen, Hong School of Information Renmin University of China Beijing China Key Laboratory of Data Engineering and Knowledge Engineering MOE China Engineering Research Center of Database and Business Intelligence MOE China Tencent AI Lab Tencent Beijing China School of Computer Science and Technology Xi'an Jiaotong University Xi'An China

Structured pruning is a widely used technique for reducing the size of pre-trained language models (PLMs), but current methods often overlook the potential of compressing the hidden dimension (d) in PLMs, a dimension critical to model size and efficiency. This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (SP3), targeting the effective reduction of d by projecting features into a space defined by principal components before masking. Extensive experiments on benchmarks (GLUE and SQuAD) show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, and maintain over 96% accuracy and outperform other methods that compress d by 6% in accuracy at the same compression ratio. SP3 has also proven effective with other models, including OPT and Llama. Our data and code are availab.e at ours repo. © 2023, CC BY.

关键词：

QUANTUM LOVÁSZ LOCAL LEMMA: SHEARER’S BOUND IS TIGHT

学校读者我要写书评

暂无评论

arXiv 2018年

作者： He, Kun Li, Qian Sun, Xiaoming Zhang, Jiapeng The Key Lab of Data Engineering and Knowledge Engineering MOE Renmin University of China Beijing China Institute of Computing Technology Chinese Academy of Sciences University of Chinese Academy of Sciences Beijing China University of Southern California United States

The Lovász Local Lemma (LLL) is a very powerful tool in combinatorics and probability theory to show the possibility of avoiding all bad events under some weakly dependent conditions. In a seminal paper, Ambainis, Kempe, and Sattath (JACM 2012) introduced a quantum version LLL (QLLL) which shows the possibility of avoiding all "bad" Hamiltonians under some weakly dependent condition, and applied QLLL to the random k-QSAT problem. Sattath, Morampudi, Laumann, and moessner (PNAS 2015) extended Ambainis, Kempe, and Sattath’s result and showed that Shearer’s bound is a sufficient condition for QLLL, and conjectured that Shearer’s bound is indeed the tight condition for QLLL. In this paper, we affirm this conjecture. Precisely, we prove that Shearer’s bound is tight for QLLL, i.e., the relative dimension of the smallest satisfying subspace is completely characterized by the independent set polynomial. Our result implies the tightness of Gilyén and Sattath’s algorithm (FOCS 2017), and also implies that the lattice gas partition function fully characterizes quantum satisfiability for almost all Hamiltonians with large enough qudits (Sattath, Morampudi, Laumann and moessner, PNAS 2015). The commuting LLL (CLLL), which focuses on commuting local Hamiltonians, is also investigated here. We prove that the tight regions of CLLL and QLLL are different in general. This result indicates that it is possible to design an algorithm for CLLL which is still efficient beyond Shearer’s bound. Copyright © 2018, The Authors. All rights reserved.

关键词： Hamiltonians

SAMPLING LOVÁSZ LOCAL LEMMA FOR GENERAL CONSTRAINT SATISFACTION SOLUTIONS IN NEAR-LINEAR TIME

学校读者我要写书评

暂无评论

arXiv 2022年

作者： He, Kun Wang, Chunyang Yin, Yitong The Key Lab of Data Engineering and Knowledge Engineering MOE Renmin University of China No. 59 Zhongguancun Street Haidian District Beijing China State Key Laboratory for Novel Software Technology Nanjing University 163 Xianlin Avenue Jiangsu Province Nanjing China

We give a fast algorithm for sampling uniform solutions of general constraint satisfaction problems (CSPs) in a local lemma regime. Suppose that the CSP has n variables with domain size at most q, each constraint contains at most k variables, shares variables with at most Δ constraints, and is violated with probability at most p by a uniform random assignment. The algorithm returns an almost uniform satisfying assignment in expected poly(q,k, Δ) ·(n) time, as long as a local lemma condition is satisfied: k · p · q2 · Δ5 ≤ C0 for a suitably small absolute constant C0. Previously, under similar local lemma conditions, sampling algorithms with running time polynomial in both nand Δ were only known for the almost atomic case, where each constraint is violated by a small number of forbidden local configurations. The key term Δ5 in our local lemma condition also improves the previously best known Δ7 for general CSPs [JPV21b] and Δ5.714 for atomic CSPs, including the special case of k-CNF [JPV21a, HSW21]. Our sampling approach departs from previous fast algorithms for sampling LLL, which were based on Markov chains. A crucial step of our algorithm is a recursive marginal sampler that is of independent interests. Within a local lemma regime, this marginal sampler can draw a random value for a variable according to its marginal distribution, at a cost independent of the size of the CSP. Copyright © 2022, The Authors. All rights reserved.

关键词： Constraint satisfaction problems

Chinese Emergency Event Extraction Based on Contrastive Learning with Event Semantic Features 19

学校读者我要写书评

暂无评论

Chinese Emergency Event Extraction Based on Contrastive Lear...

19th International Conference on Natural Computation, Fuzzy Systems and knowledge Discovery, ICNC-FSKD 2023

作者： Sheng, Xinyi Gu, Jinguang Yan, Youcheng Xu, Fangfang Wuhan University of Science and Technology College of Computer Science and Technology Wuhan430065 China Institute of Big Data Science and Engineering Wuhan University of Science and Technology Wuhan430065 China The Key Lab. of Rich-Media Knowledge Org. and Serv. of Digit. Publ. Content Inst. of Sci. and Tech. Info. of China Beijing100038 China

ISBN: (纸本)9798350304398

Extracting emergency events from a large amount of unstructured information is essential for improving early warning and emergency response. Existing event extraction methods for specialist fields often rely on well-designed syntactic dependencies or external knowledge graphs, which can provide high accuracy in their specialist fields. However, these approaches may have poor generalization capabilities and are difficult to apply directly to the emergency field. In addition, most existing methods ignore the interdependencies between event triggers and event arguments, which may be crucial semantic features for event extraction. To solve these problems, we propose a Chinese emergent event extraction method based on contrastive learning with event semantic features(CNEmergEE). Firstly, the proposed approach automatically expands the pre-trained model's vocabulary on a domain corpus. Then, continuing learning on the same corpus allows the model to complete domain adaptation. Finally, we use contrast learning to enable models to learn semantic representations of events. Experimental results on the CEC (Chinese Emergency Corpus) dataset show that our approach achieves better improvement. © 2023 IEEE.

关键词： Semantics

Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniques 38

学校读者我要写书评

暂无评论

Suppress Content Shift: Better Diffusion Features via Off-th...

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Meng, Benyuan Xu, Qianqian Wang, Zitai Yang, Zhiyong Cao, Xiaochun Huang, Qingming Institute of Information Engineering CAS China School of Cyber Security University of Chinese Academy of Sciences China Key Lab. of Intelligent Information Processing Institute of Computing Technology CAS China Peng Cheng Laboratory China School of Computer Science and Tech. University of Chinese Academy of Sciences China School of Cyber Science and Tech. Shenzhen Campus of Sun Yat-sen University China Key Laboratory of Big Data Mining and Knowledge Management CAS China

Diffusion models are powerful generative models, and this capability can also be applied to discrimination. The inner activations of a pre-trained diffusion model can serve as features for discriminative tasks, namely, diffusion feature. We discover that diffusion feature has been hindered by a hidden yet universal phenomenon that we call content shift. To be specific, there are content differences between features and the input image, such as the exact shape of a certain object. We locate the cause of content shift as one inherent characteristic of diffusion models, which suggests the broad existence of this phenomenon in diffusion feature. Further empirical study also indicates that its negative impact is not negligible even when content shift is not visually perceivable. Hence, we propose to suppress content shift to enhance the overall quality of diffusion features. Specifically, content shift is related to the information drift during the process of recovering an image from the noisy input, pointing out the possibility of turning off-the-shelf generation techniques into tools for content shift suppression. We further propose a practical guideline named GATE to efficiently evaluate the potential benefit of a technique and provide an implementation of our methodology. Despite the simplicity, the proposed approach has achieved superior results on various tasks and datasets, validating its potential as a generic booster for diffusion features. Our code is availab.e at this url. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features 38

学校读者我要写书评

暂无评论

Not All Diffusion Model Activations Have Been Evaluated as D...

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Meng, Benyuan Xu, Qianqian Wang, Zitai Cao, Xiaochun Huang, Qingming Institute of Information Engineering CAS China School of Cyber Security University of Chinese Academy of Sciences China Key Lab. of Intelligent Information Processing Institute of Computing Technology CAS China Peng Cheng Laboratory China School of Cyber Science and Tech. Shenzhen Campus of Sun Yat-sen University China School of Computer Science and Tech. University of Chinese Academy of Sciences China Key Laboratory of Big Data Mining and Knowledge Management CAS China

Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting a small yet effective subset poses a fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advancements in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes a further step with a much broader range of activations evaluated. Considering the significant increase in activations, a full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activations that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is availab.e at this url. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

Learning Two-Stream CNN for Multi-Modal Age-related Macular Degeneration Categorization

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Wang, Weisen Li, Xirong Xu, Zhiyan Yu, Weihong Zhao, Jianchun Ding, Dayong Chen, Youxin The MoE Key Lab of Data Engineering and Knowledge Engineering Renmin University of China Beijing100872 China Aimc Lab School of Information Renmin University of China Beijing100872 China Vistel Ai Lab Visionary Intelligence Ltd. Beijing100872 China The Key Lab of Ocular Fundus Disease Chinese Academy of Medical Sciences Dept. of Ophthalmology Peking Union Medical College Hospital Beijing100730 China

This paper tackles automated categorization of Agerelated Macular Degeneration (AMD), a common macular disease among people over 50. Previous research efforts mainly focus on AMD categorization with a single-modal input, let it be a color fundus photograph (CFP) or an OCT B-scan image. By contrast, we consider AMD categorization given a multi-modal input, a direction that is clinically meaningful yet mostly unexplored. Contrary to the prior art that takes a traditional approach of feature extraction plus classifier training that cannot be jointly optimized, we opt for end-to-end multi-modal Convolutional Neural Networks (MM-CNN). Our MM-CNN is instantiated by a two-stream CNN, with spatially-invariant fusion to combine information from the CFP and OCT streams. In order to visually interpret the contribution of the individual modalities to the final prediction, we extend the class activation mapping (CAM) technique to the multi-modal scenario. For effective training of MM-CNN, we develop two data augmentation methods. One is GAN-based CFP/OCT image synthesis, with our novel use of CAMs as conditional input of a high-resolution image-to-image translation GAN. The other method is Loose Pairing, which pairs a CFP image and an OCT image on the basis of their classes instead of eye identities. Experiments on a clinical dataset consisting of 1,094 CFP images and 1,289 OCT images acquired from 1,093 distinct eyes show that the proposed solution obtains better F1 and Accuracy than multiple baselines for multimodal AMD categorization. Code and data are availab.e at https://***/li-xirong/mmc-amd. Copyright © 2020, The Authors. All rights reserved.

关键词： Convolutional neural networks

QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Fu, Weiping Wei, Bifan Hu, Jianxiang Cai, Zhongmin Liu, Jun School of Computer Science and Technology Xi’an Jiaotong University Xi’an China School of Continuing Education Xi’an Jiaotong University Xi’an China MOE KLINNS Lab School of Automation Science and Engineering Xi’an Jiaotong University Xi’an China Shaanxi Province Key Laboratory of Big Data Knowledge Engineering Xi’an Jiaotong University Xi’an China

Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies, requiring a reliable and comprehensive evaluation of their quality. Human evaluation is widely used in the field of question generation (QG) and serves as the gold standard for automatic metrics. However, there is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics. To address this, we propose QGEval, a multi-dimensional Evaluation benchmark for Question Generation, which evaluates both generated questions and existing automatic metrics across 7 dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. We demonstrate the appropriateness of these dimensions by examining their correlations and distinctions. Through consistent evaluations of QG models and automatic metrics with QGEval, we find that 1) most QG models perform unsatisfactorily in terms of answerability and answer consistency, and 2) existing metrics fail to align well with human judgments when evaluating generated questions across the 7 dimensions. We expect this work to foster the development of both QG technologies and their evaluation. Copyright © 2024, The Authors. All rights reserved.

关键词： Benchmarking

Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Dong, Jianfeng Wang, Yabing Chen, Xianke Qu, Xiaoye Li, Xirong He, Yuan Wang, Xun The College of Computer and Information Engineering Zhejiang Gongshang University Hangzhou310035 China The State Key Laboratory of Information Security Institute of Information Engineering Chinese Academy of Sciences Beijing100093 China The School of Electronic Information and Communications Huazhong University of Science and Technology Hubei430074 China The Key Lab of Data Engineering and Knowledge Engineering Renmin University of China The AIMC Lab. School of Information Renmin University of China Beijing100872 China The Alibaba Group Beijing100102 China

This paper aims for the task of text-to-video retrieval, where given a query in the form of a natural-language sentence, it is asked to retrieve videos which are semantically relevant to the given query, from a great number of unlab.led videos. The success of this task depends on cross-modal representation learning that projects both videos and sentences into common spaces for semantic similarity computation. In this work, we concentrate on video representation learning, an essential component for text-to-video retrieval. Inspired by the reading strategy of humans, we propose a Reading-strategy Inspired Visual Representation Learning (RIVRL) to represent videos, which consists of two branches: a previewing branch and an intensive-reading branch. The previewing branch is designed to briefly capture the overview information of videos, while the intensive-reading branch is designed to obtain more in-depth information. Moreover, the intensive-reading branch is aware of the video overview captured by the previewing branch. Such holistic information is found to be useful for the intensive-reading branch to extract more fine-grained features. Extensive experiments on three datasets are conducted, where our model RIVRL achieves a new state-of-the-art on TGIF and VATEX. Moreover, on MSR-VTT, our model using two video features shows comparable performance to the state-of-the-art using seven video features and even outperforms models pre-trained on the large-scale HowTo100M dataset. Code is availab.e at https://***/LiJiaBei-7/rivrl. © 2022, CC BY.

关键词： Video recording