检索结果-内蒙古大学图书馆

Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Li, Yingji Du, Mengnan Wang, Xin Wang, Ying College of Computer Science and Technology Jilin University Changchun China Department of Data Science New Jersey Institute of Technology Newark United States School of Artificial Intelligence Jilin University Changchun China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University Changchun China

As the representation capability of Pre-trained Language Models (PLMs) improve, there is growing concern that they will inherit social biases from unprocessed corpora. Most previous debiasing techniques used Counterfactual data Augmentation (CDA) to balance the training corpus. However, CDA slightly modifies the original corpus, limiting the representation distance between different demographic groups to a narrow range. As a result, the debiasing model easily fits the differences between counterfactual pairs, which affects its debiasing performance with limited text resources. In this paper, we propose an adversarial training-inspired two-stage debiasing model using Contrastive learning with Continuous Prompt Augmentation (named CCPA) to mitigate social biases in PLMs’ encoding. In the first stage, we propose a data augmentation method based on continuous prompt tuning to push farther the representation distance between sample pairs along different demographic groups. In the second stage, we utilize contrastive learning to pull closer the representation distance between the augmented sample pairs and then fine-tune PLMs’ parameters to get debiased encoding. Our approach guides the model to achieve stronger debiasing performance by adding difficulty to the training process. Extensive experiments show that CCPA outperforms baselines in terms of debiasing performance. Meanwhile, experimental results on the GLUE benchmark show that CCPA retains the language modeling capability of PLMs. Copyright © 2023, The Authors. All rights reserved.

关键词： Encoding (symbols)

ReconBoost: Boosting Can Achieve Modality Reconcilement

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hua, Cong Xu, Qianqian Bao, Shilong Yang, Zhiyong Huang, Qingming Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences Beijing China School of Computer Science and Technology University of Chinese Academy of Sciences Beijing China Institute of Information Engineering Chinese Academy of Sciences Beijing China School of Cyber Security University of Chinese Academy of Sciences Beijing China Key Laboratory of Big Data Mining and Knowledge Management Chinese Academy of Sciences Beijing China

This paper explores a novel multi-modal alternating learning paradigm pursuing a reconciliation between the exploitation of uni-modal features and the exploration of cross-modal interactions. This is motivated by the fact that current paradigms of multi-modal learning tend to explore multi-modal features simultaneously. The resulting gradient prohibits further exploitation of the features in the weak modality, leading to modality competition, where the dominant modality overpowers the learning process. To address this issue, we study the modality-alternating learning paradigm to achieve reconcilement. Specifically, we propose a new method called ReconBoost to update a fixed modality each time. Herein, the learning objective is dynamically adjusted with a reconcilement regularization against competition with the historical models. By choosing a KL-based reconcilement, we show that the proposed method resembles Friedman's Gradient-Boosting (GB) algorithm, where the updated learner can correct errors made by others and help enhance the overall performance. The major difference with the classic GB is that we only preserve the newest model for each modality to avoid overfitting caused by ensembling strong learners. Furthermore, we propose a memory consolidation scheme and a global rectification scheme to make this strategy more effective. Experiments over six multi-modal benchmarks speak to the efficacy of the method. We release the code at https://***/huacong/ReconBoost. Copyright © 2024, The Authors. All rights reserved.

关键词： Machine learning

JC5A: Service Delay Minimization for Aerial MEC-assisted Industrial Cyber-Physical Systems

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Sun, Geng Wu, Jiaxu Sun, Zemin He, Long Wang, Jiacheng Niyato, Dusit Jamalipour, Abbas Mao, Shiwen College of Computer Science and Technology Jilin University Changchun130012 China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University Changchun130012 China College of Computing and Data Science Nanyang Technological University Singapore639798 Singapore School of Computer Science and Engineering Nanyang Technological University Singapore639798 Singapore School of Electrical and Computer Engineering The University of Sydney SydneyNSW2006 Australia Department of Electrical and Computer Engineering Auburn University Auburn United States

In the era of the sixth generation (6G) and industrial Internet of Things (IIoT), an industrial cyber-physical system (ICPS) drives the proliferation of sensor devices and computing-intensive tasks. To address the limited resources of IIoT sensor devices, unmanned aerial vehicle (UAV)-assisted mobile edge computing (MEC) has emerged as a promising solution, providing flexible and cost-effective services in close proximity of IIoT sensor devices (ISDs). However, leveraging aerial MEC to meet the delay-sensitive and computation-intensive requirements of the ISDs could face several challenges, including the limited communication, computation and caching (3C) resources, stringent offloading requirements for 3C services, and constrained on-board energy of UAVs. To address these issues, we first present a collaborative aerial MEC-assisted ICPS architecture by incorporating the computing capabilities of the macro base station (MBS) and UAVs. We then formulate a service delay minimization optimization problem (SDMOP). Since the SDMOP is proved to be an NP-hard problem, we propose a joint computation offloading, caching, communication resource allocation, computation resource allocation, and UAV trajectory control approach (JC5A). Specifically, JC5A consists of a block successive upper bound minimization method of multipliers (BSUMM) for computation offloading and service caching, a convex optimization-based method for communication and computation resource allocation, and a successive convex approximation (SCA)-based method for UAV trajectory control. Moreover, we theoretically prove the convergence and polynomial complexity of JC5A. Simulation results demonstrate that the proposed approach can achieve superior system performance compared to the benchmark approaches and algorithms. Copyright © 2024, The Authors. All rights reserved.

关键词： Convex optimization

An Empirical Study on Information Extraction using Large Language Models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Han, Ridong Yang, Chaohao Peng, Tao Tiwari, Prayag Wan, Xiang Liu, Lu Wang, Benyou College of Computer Science and Technology Jilin University China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University China College of Software Jilin University China Shenzhen Research Institute of Big Data The Chinese University of Hong Kong Shenzhen China School of Data Science The Chinese University of Hong Kong Shenzhen China School of Information Technology Halmstad University Sweden

Human-like large language models (LLMs), especially the most powerful and popular ones in OpenAI’s GPT family, have proven to be very helpful for many natural language processing (NLP) related tasks. Therefore, various attempts have been made to apply LLMs to information extraction (IE), which is a fundamental NLP task that involves extracting information from unstructured plain text. To demonstrate the latest representative progress in LLMs’ information extraction ability, we assess the information extraction ability of GPT-4 (the latest version of GPT at the time of writing this paper) from four perspectives: Performance, Evaluation Criteria, Robustness, and Error Types. Our results suggest a visible performance gap between GPT-4 and state-of-the-art (SOTA) IE methods. To alleviate this problem, considering the LLMs’ human-like characteristics, we propose and analyze the effects of a series of simple prompt-based methods, which can be generalized to other LLMs and NLP tasks. Rich experiments show our methods’ effectiveness and some of their remaining issues in improving GPT-4’s information extraction ability. Copyright © 2023, The Authors. All rights reserved.

关键词： Information retrieval

Recovering accurate labeling information from partially valid data for effective multi-label learning 29

学校读者我要写书评

暂无评论

Recovering accurate labeling information from partially vali...

29th International Joint Conference on Artificial Intelligence, IJCAI 2020

作者： Li, Ximing Wang, Yang College of Computer Science and Technology Jilin University China Key Laboratory of Symbolic Computation Knowledge Engineering of Ministry of Education Jilin University China Key Laboratory of Knowledge Engineering With Big Data Ministry of Education Hefei University of Technology China School of Computer Sci and Information Engineering Hefei University of Technology China

ISBN: (纸本)9780999241165

Partial Multi-label Learning (PML) aims to induce the multi-label predictor from datasets with noisy supervision, where each training instance is associated with several candidate labels but only partially valid. To address the noisy issue, the existing PML methods basically recover the ground-truth labels by leveraging the ground-truth confidence of the candidate label, i.e., the likelihood of a candidate label being a ground-truth one. However, they neglect the information from non-candidate labels, which potentially contributes to the ground-truth label recovery. In this paper, we propose to recover the ground-truth labels, i.e., estimating the ground-truth confidences, from the label enrichment, composed of the relevance degrees of candidate labels and irrelevance degrees of non-candidate labels. Upon this observation, we further develop a novel two-stage PML method, namely Partial MultiLabel Learning with Label Enrichment-Recovery (PML3ER), where in the first stage, it estimates the label enrichment with unconstrained label propagation, then jointly learns the ground-truth confidence and multi-label predictor given the label enrichment. Experimental results validate that PML3ER outperforms the state-of-the-art PML methods. © 2020 Inst. Sci. inf., Univ. Defence in Belgrade. All rights reserved.

关键词： Learning systems

Evaluating the Performance to Detecting Topic Shift of Large Language Models: A Study

学校读者我要写书评

暂无评论

Evaluating the Performance to Detecting Topic Shift of Large...

6th International Conference on Intelligent Computing and Signal Processing (ICSP)

作者： Pingfang Tian Deting Liu Department of Computer Science and Technology Wuhan University of Science and Technology Wuhan China The Key Laboratory of the Rich Media Knowledge Organisation and Service of Digital Publishing Content Institute of Scientific and Technical Information of China Beijing China Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System Wuhan China Big Data Science and Engineering Research Institute Wuhan University of Science and Technology Wuhan China

ISBN: (数字)9798350376548

ISBN: (纸本)9798350376555

Topic shift detection aims to identify whether there is a change in the current topic of conversation or if a change is needed. The study found previous work did not evaluate the performance of large language models like ChatGPT on the task of topic shift. Therefore, this paper's main task and innovation lie in analyzing ChatGPT's performance on topic shift detection. To provide a more comprehensive evaluation, we conducted topic shift detection tasks on ChatGPT from three aspects: single utterances, adjacent utterances, and contextual levels. Additionally, to gauge the performance of large language models, we conducted experiments on multiple small-scale models and compared the results of the two models. Experimental results on the publicly available English TIAGE dataset showed that small-scale models exhibited lower recall in all three aspects, while ChatGPT performed better in the recall. This suggests that compared to small-scale models, large models are more capable of accurately detecting topic shifts. However, large models also exhibited lower precision, indicating that while ChatGPT can recognize content differences in utterances, its judgment on whether these different contents belong to the same topic is poor.

关键词： Technological innovation Large language models Oral communication Signal processing Chatbots Context modeling

Search Result Diversification Framework Based on Dual Star-shaped Self-Attention Network 20

学校读者我要写书评

暂无评论

Search Result Diversification Framework Based on Dual Star-s...

20th Chinese National Conference on Computational Linguistics, CCL 2021

作者： Qin, Xubo Dou, Zhicheng Zhu, Yutao Wen, Ji-Rong School of Information Renmin University Gaoling School of Artificial Intelligence Renmin University of China Université de Montréal Beijing Key Laboratory of Big Data Management and Analysis Methods Key Laboratory of Data Engineering and Knowledge Engineering MOE

Research shows that most queries issued by users are short queries. Those queries may be ambiguous or broad for specifying users' actual information need. The same query issued by different users may lead to different information needs. In order to satisfy different users' needs, the search engines need to rank the search result document diversely. This leads to the search result diversification technology. For those previous diversification approaches based on global document interactions, fully-connected self-attention networks are used for capturing the interactions among all the candidate documents, leading to well performance. However, those models only take the document interactions into consideration, ignoring the relevance signals of candidate documents. As the annotated datasets are not enough for model training, the effectiveness of the model will be limited. In this paper we prove a new implicit diversification framework based on self-attention network in the shape of dual-star. A star-shaped topological structure is deployed with an external query node, in order to effectively extract the global interaction signals relevant to the query. Using self-attention to learn the similarity between each document, this model can be trained easily, and perform significantly better than the state-of-the-art implicit approach indicating the potential of implicit methods. © 2021 China National Conference on Computational Linguistics Published under Creative Commons Attribution 4.0 International License

关键词： Search engines

Injecting Numerical Reasoning Skills into knowledge Base Question Answering Models

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Feng, Yu Zhang, Jing Zhang, Xiaokang Liu, Lemao Li, Cuiping Chen, Hong Key Laboratory of Data Engineering and Knowledge Engineering Ministry of Education China School of Information Renmin University of China China Tencent AI Lab United States

Embedding-based methods are popular for knowledge Base Question Answering (KBQA), but few current models have numerical reasoning skills and thus struggle to answer ordinal constrained questions. This paper proposes a new embedding-based KBQA framework which particularly takes numerical reasoning into account. We present NumericalTransformer on top of NSM, a state-of-the-art embedding-based KBQA model, to create NT-NSM. To enable better training, we propose two pre-training tasks with explicit numerical-oriented loss functions on two generated training datasets and a template-based data augmentation method for enriching ordinal constrained QA dataset. Extensive experiments on KBQA benchmarks demonstrate that with the help of our training algorithm, NT-NSM is empowered with numerical reasoning skills and substantially outperforms the baselines in answering ordinal constrained questions. Copyright © 2021, The Authors. All rights reserved.

关键词： Embeddings

Aerial Reliable Collaborative Communications for Terrestrial Mobile Users via Evolutionary Multi-Objective Deep Reinforcement Learning

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Sun, Geng Xiao, Jian Li, Jiahui Wang, Jiacheng Kang, Jiawen Niyato, Dusit Mao, Shiwen College of Computer Science and Technology Jilin University Changchun130012 China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University Changchun130012 China affiliated with the College of Computing and Data Science Nanyang Technological University Singapore639798 Singapore School of Computer Science and Engineering Nanyang Technological University Singapore639798 Singapore School of Automation Guangdong University of Technology Guangzhou510641 China Department of Electrical and Computer Engineering Auburn University AuburnAL36849-5201 United States

Unmanned aerial vehicles (UAVs) have emerged as the potential aerial base stations (BSs) to improve terrestrial communications. However, the limited onboard energy and antenna power of a UAV restrict its communication range and transmission capability. To address these limitations, this work employs collaborative beamforming through a UAV-enabled virtual antenna array to improve transmission performance from the UAV to terrestrial mobile users, under interference from non-associated BSs and dynamic channel conditions. Specifically, we introduce a memory-based random walk model to more accurately depict the mobility patterns of terrestrial mobile users. Following this, we formulate a multi-objective optimization problem (MOP) focused on maximizing the transmission rate while minimizing the flight energy consumption of the UAV swarm. Given the NP-hard nature of the formulated MOP and the highly dynamic environment, we transform this problem into a multi-objective Markov decision process and propose an improved evolutionary multi-objective reinforcement learning algorithm. Specifically, this algorithm introduces an evolutionary learning approach to obtain the approximate Pareto set for the formulated MOP. Moreover, the algorithm incorporates a long short-term memory network and hyper-sphere-based task selection method to discern the movement patterns of terrestrial mobile users and improve the diversity of the obtained Pareto set. Simulation results demonstrate that the proposed method effectively generates a diverse range of non-dominated policies and outperforms existing methods. Additional simulations demonstrate the scalability and robustness of the proposed CB-based method under different system parameters and various unexpected circumstances. © 2025, CC0.

关键词： Unmanned aerial vehicles (UAV)