检索结果-内蒙古大学图书馆

Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Liang, Xinnian Zhou, Zefan Huang, Hui Wu, Shuangzhi Xiao, Tong Yang, Muyun Li, Zhoujun Bian, Chao State Key Lab of Software Development Environment Beihang University Beijing China School of Computer Science and Engineering Northeastern University Shenyang China Faculty of Computing Harbin Institute of Technology Harbin China Lark Platform Engineering-AI Beijing China

Pretrained language models (PLMs) have shown marvelous improvements across various NLP tasks. Most Chinese PLMs simply treat an input text as a sequence of characters, and completely ignore word information. Although Whole Word Masking can alleviate this, the semantics in words is still not well represented. In this paper, we revisit the segmentation granularity of Chinese PLMs. We propose a mixed-granularity Chinese BERT (MigBERT) by considering both characters and words. To achieve this, we design objective functions for learning both character and word-level representations. We conduct extensive experiments on various Chinese NLP tasks to evaluate existing PLMs as well as the proposed MigBERT. Experimental results show that MigBERT achieves new SOTA performance on all these tasks. Further analysis demonstrates that words are semantically richer than characters. More interestingly, we show that MigBERT also works with Japanese. Our code has been released here 1 and you can download our model here 2 © 2023, CC BY.

关键词： Semantics

Multi-View Incongruity Learning for Multimodal Sarcasm Detection

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Guo, Diandian Cao, Cong Yuan, Fangfang Liu, Yanbing Zeng, Guangjie Yu, Xiaoyan Peng, Hao Yu, Philip S. Institute of Information Engineering Chinese Academy of Sciences China School of Cyber Security University of Chinese Academy of Sciences China State Key Laboratory of Software Development Environment Beihang University China School of Computer Science and Technology Beijing Institute of Technology China Department of Computer Science University of Illinois Chicago United States

Multimodal sarcasm detection (MSD) is essential for various downstream tasks. Existing MSD methods tend to rely on spurious correlations. These methods often mistakenly prioritize non-essential features yet still make correct predictions, demonstrating poor generalizability beyond training environments. Regarding this phenomenon, this paper undertakes several initiatives. Firstly, we identify two primary causes that lead to the reliance of spurious correlations. Secondly, we address these challenges by proposing a novel method that integrate Multimodal Incongruities via Contrastive Learning (MICL) for multimodal sarcasm detection. Specifically, we first leverage incongruity to drive multi-view learning from three views: token-patch, entity-object, and sentiment. Then, we introduce extensive data augmentation to mitigate the biased learning of the textual modality. Additionally, we construct a test set, SPMSD, which consists potential spurious correlations to evaluate the the model’s generalizability. Experimental results demonstrate the superiority of MICL on benchmark datasets, along with the analyses showcasing MICL’s advancement in mitigating the effect of spurious correlation. Copyright © 2024, The Authors. All rights reserved.

关键词： Contrastive Learning

Container lifecycle-aware scheduling for serverless computing

学校读者我要写书评

暂无评论

Container lifecycle-aware scheduling for serverless computin...

作者： Wu, Song Tao, Zhiheng Fan, Hao Huang, Zhuo Zhang, Xinmin Jin, Hai Yu, Chen Cao, Chun National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab Cluster and Grid Computing Lab School of Computer Science and Technology Huazhong University of Science and Technology Wuhan China State Key Laboratory for Novel Software Technology Department of Computer Science and Technology Nanjing University Nanjing China

Elastic scaling in response to changes on demand is a main benefit of serverless computing. When bursty workloads arrive, a serverless platform launches many new containers and initializes function environments (known as cold starts), which incurs significant startup latency. To reduce cold starts, platforms usually pause a container after it serves a request, and reuse this container for subsequent requests. However, this reuse strategy cannot efficiently reduce cold starts because the schedulers are agnostic of container lifecycle. For example, it may ignore soon available containers or evict soon needed containers. We propose a container lifecycle-aware scheduling strategy for serverless computing, CAS. The key idea is to control distribution of requests and determine creation or eviction of containers according to different lifecycle phases of containers. We implement a prototype of CAS on OpenWhisk. Our evaluation shows that CAS reduces 81% cold starts and therefore brings a 63% reduction at 95th percentile latency compared with native scheduling strategy in OpenWhisk when there is worker contention between workloads, and does not add significant performance overhead. © 2021 John Wiley & Sons Ltd.

关键词： Containers

Time-Aware Missing Traffic Flow Prediction for Sensors with Privacy-Preservation 11th

学校读者我要写书评

暂无评论

Time-Aware Missing Traffic Flow Prediction for Sensors with ...

11th International Conference on computer engineering and Networks, CENet2021

作者： Qi, Lianyong Wang, Fan Xu, Xiaolong Dou, Wanchun Zhang, Xuyun Khosravi, Mohammad R. Zhou, Xiaokang School of Computer Science Qufu Normal University Qufu China School of Computer and Software Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology Nanjing University of Information Science and Technology Nanjing China State Key Laboratory for Novel Software Technology Department of Computer Science and Technology Nanjing University Nanjing China Department of Computing Macquarie University SydneyNSW2109 Australia Department of Computer Engineering Persian Gulf University Bushehr Iran Department of Electrical and Electronic Engineering Shiraz University of Technology Shiraz Iran Faculty of Data Science Shiga University Hikone Japan

ISBN: (纸本)9789811665530

With the continuous development of IoT, a number of sensors establish on the roadside to monitor traffic conditions in real time. The continuously traffic data generated by these sensors makes traffic management feasible. However, loss of data may occur due to inevitable sensor failure, impeding traffic managers to understand traffic dynamics clearly. In this situation, it is becoming a necessity to predict missing traffic flow accurately for effective traffic management. Furthermore, the traffic sensor data are often distributed and stored by different agencies, which inhibits the multi-party sensor data sharing significantly due to privacy concerns. Therefore, it has become a major obstacle to balance the tradeoff between data sharing and vehicle privacy. In light of these challenges, we propose a privacy-aware and accurate missing traffic flow prediction approach based on time-aware Locality-Sensitive Hashing technique. At last, we deploy a set of experiments based on a real traffic dataset. Experimental reports demonstrate the feasibility of our proposal in terms of traffic flow prediction accuracy and efficiency while guaranteeing sensor data privacy. © 2022, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

关键词： Data privacy

Programming by Example Made Easy

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wu, Jiarong Wei, Lili Jiang, Yanyan Cheung, Shing-Chi Ren, Luyao Xu, Chang Department of Computer Science and Engineering Hong Kong University of Science and Technology Hong Kong Department of Electrical and Computer Engineering McGill University Montreal Canada State Key Laboratory for Novel Software Technology Department of Computer Science and Technology Nanjing University Nanjing China Key Lab of High Confidence Software Technologies Ministry of Education Department of Computer Science and Technology EECS Peking University Beijing China

Programming by example (PBE) is an emerging programming paradigm that automatically synthesizes programs specified by user-provided input-output examples. Despite the convenience for end-users, implementing PBE tools often requires strong expertise in programming language and synthesis algorithms. Such a level of knowledge is uncommon among software developers. It greatly limits the broad adoption of PBE by the industry. To facilitate the adoption of PBE techniques, we propose a PBE framework called Bee, which leverages an "entity-action" model based on relational tables to ease PBE development for a wide but restrained range of domains. Implementing PBE tools with Bee only requires adapting domain-specific data entities and user actions to tables, with no need to design a domain-specific language or an efficient synthesis algorithm. The synthesis algorithm of Bee exploits bidirectional searching and constraint-solving techniques to address the challenge of value computation nested in table transformation. We evaluated Bee’s effectiveness on 64 PBE tasks from three different domains and usability with a human study of 12 participants. Evaluation results show that Bee is easier to learn and use than the state-of-the-art PBE framework, and the bidirectional algorithm achieves comparable performance to domain-specifically optimized synthesizers. Copyright © 2023, The Authors. All rights reserved.

关键词： Problem oriented languages

Improving variational autoencoders with density gap-based regularization 22

学校读者我要写书评

暂无评论

Improving variational autoencoders with density gap-based re...

Proceedings of the 36th International Conference on Neural Information Processing Systems

作者： Jianfei Zhang Jun Bai Chenghua Lin Yanmeng Wang Wenge Rong State Key Laboratory of Software Development Environment Beihang University China and School of Computer Science and Engineering Beihang University China Department of Computer Science University of Sheffield United Kingdom Ping An Technology China

ISBN: (纸本)9781713871088

Variational autoencoders (VAEs) are one of the most powerful unsupervised learning frameworks in NLP for latent representation learning and latent-directed generation. The classic optimization goal of VAEs is to maximize the Evidence Lower Bound (ELBo), which consists of a conditional likelihood for generation and a negative Kullback-Leibler (KL) divergence for regularization. In practice, optimizing ELBo often leads the posterior distribution of all samples converging to the same degenerated local optimum, namely posterior collapse or KL vanishing. There are effective ways proposed to prevent posterior collapse in VAEs, but we observe that they in essence make trade-offs between posterior collapse and the hole problem, i.e., the mismatch between the aggregated posterior distribution and the prior distribution. To this end, we introduce new training objectives to tackle both problems through a novel regularization based on the probabilistic density gap between the aggregated posterior distribution and the prior distribution. Through experiments on language modeling, latent space visualization, and interpolation, we show that our proposed method can solve both problems effectively and thus outperforms the existing methods in latent-directed generation. To the best of our knowledge, we are the first to jointly solve the hole problem and posterior collapse.

关键词：

Improving Variational Autoencoders with Density Gap-based Regularization

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhang, Jianfei Bai, Jun Lin, Chenghua Wang, Yanmeng Rong, Wenge State Key Laboratory of Software Development Environment Beihang University China School of Computer Science and Engineering Beihang University China Department of Computer Science University of Sheffield United Kingdom Ping An Technology China

关键词： Economic and social effects

Simple Image-level Classification Improves Open-vocabulary Object Detection

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Fang, Ruohuan Pang, Guansong Bai, Xiao School of Computer Science and Engineering Beihang University China School of Computing and Information Systems Singapore Management University Singapore State Key Laboratory of Software Development Environment Jiangxi Research Institute Beihang University China

Open-Vocabulary Object Detection (OVOD) aims to detect novel objects beyond a given set of base categories on which the detection model is trained. Recent OVOD methods focus on adapting the image-level pre-trained vision-language models (VLMs), such as CLIP, to a region-level object detection task via, e.g., region-level knowledge distillation, regional prompt learning, or region-text pre-training, to expand the detection vocabulary. These methods have demonstrated remarkable performance in recognizing regional visual concepts, but they are weak in exploiting the VLMs' powerful global scene understanding ability learned from the billion-scale image-level text descriptions. This limits their capability in detecting hard objects of small, blurred, or occluded appearance from novel/base categories, whose detection heavily relies on contextual information. To address this, we propose a novel approach, namely Simple Image-level Classification for Context-Aware Detection Scoring (SIC-CADS), to leverage the superior global knowledge yielded from CLIP for complementing the current OVOD models from a global perspective. The core of SIC-CADS is a multi-modal multi-label recognition (MLR) module that learns the object co-occurrence-based contextual information from CLIP to recognize all possible object categories in the scene. These image-level MLR scores can then be utilized to refine the instance-level detection scores of the current OVOD models in detecting those hard objects. This is verified by extensive empirical results on two popular benchmarks, OV-LVIS and OV-COCO, which show that SIC-CADS achieves significant and consistent improvement when combined with different types of OVOD models. Further, SIC-CADS also improves the cross-dataset generalization ability on Objects365 and OpenImages. Code is available at https://***/mala-lab/SIC-CADS. Copyright © 2023, The Authors. All rights reserved.

关键词： Object detection

DeepRicci: Self-supervised Graph Structure-Feature Co-Refinement for Alleviating Over-squashing

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Sun, Li Huang, Zhenhao Wu, Hua Ye, Junda Peng, Hao Yu, Zhengtao Yu, Philip S. School of Control and Computer Engineering North China Electric Power University Beijing102206 China School of Computer Science Beijing University of Posts and Telecommunications Beijing100876 China State Key Laboratory of Software Development Environment Beihang University Beijing100191 China Faculty of Information Engineering and Automation Kunming University of Science and Technology Kunming650500 China Department of Computer Science University of Illinois ChicagoIL United States

Graph Neural Networks (GNNs) have shown great power for learning and mining on graphs, and Graph Structure Learning (GSL) plays an important role in boosting GNNs with a refined graph. In the literature, most GSL solutions either primarily focus on structure refinement with task-specific supervision (i.e., node classification), or overlook the inherent weakness of GNNs themselves (e.g., over-squashing), resulting in suboptimal performance despite sophisticated designs. In light of these limitations, we propose to study self-supervised graph structure-feature co-refinement for effectively alleviating the issue of over-squashing in typical GNNs. In this paper, we take a fundamentally different perspective of the Ricci curvature in Riemannian geometry, in which we encounter the challenges of modeling, utilizing and computing Ricci curvature. To tackle these challenges, we present a self-supervised Riemannian model, DeepRicci. Specifically, we introduce a latent Riemannian space of heterogeneous curvatures to model various Ricci curvatures, and propose a gyrovector feature mapping to utilize Ricci curvature for typical GNNs. Thereafter, we refine node features by geometric contrastive learning among different geometric views, and simultaneously refine graph structure by backward Ricci flow based on a novel formulation of differentiable Ricci curvature. Finally, extensive experiments on public datasets show the superiority of DeepRicci, and the connection between backward Ricci flow and over-squashing. Codes of our work are given in https://***/RiemanGraph/. © 2024, CC BY-NC-SA.

关键词： Graph neural networks