检索结果-内蒙古大学图书馆

X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Swetha, Sirnam Yang, Jinyu Neiman, Tal Rizve, Mamshad Nayeem Tran, Son Yao, Benjamin Chilimbi, Trishul Shah, Mubarak Center for Research in Computer Vision University of Central Florida United States Amazon United States

Recent advancements in Multimodal Large Language Models (MLLMs) have revolutionized the field of vision-language understanding by integrating visual perception capabilities into Large Language Models (LLMs). The prevailing trend in this field involves the utilization of a vision encoder derived from vision-language contrastive learning (CL), showing expertise in capturing overall representations while facing difficulties in capturing detailed local patterns. In this work, we focus on enhancing the visual representations for MLLMs by combining high-frequency and detailed visual representations, obtained through masked image modeling (MIM), with semantically-enriched low-frequency representations captured by CL. To achieve this goal, we introduce X-Former which is a lightweight transformer module designed to exploit the complementary strengths of CL and MIM through an innovative interaction mechanism. Specifically, X-Former first bootstraps vision-language representation learning and multimodal-to-multimodal generative learning from two frozen vision encoders, i.e., CLIP-ViT (CL-based) and MAE-ViT (MIM-based). It further bootstraps vision-to-language generative learning from a frozen LLM to ensure visual features from X-Former can be interpreted by the LLM. To demonstrate the effectiveness of our approach, we assess its performance on tasks demanding detailed visual understanding. Extensive evaluations indicate that X-Former excels in visual reasoning tasks involving both structural and semantic categories in the GQA dataset. Assessment on fine-grained visual perception benchmark further confirms its superior capabilities in visual understanding. © 2024, CC BY-NC-SA.

关键词： Image enhancement

A CNN-LSTM Network for Classification of Attention Deficit Hyperactivity Disorder from EEG Data 41

学校读者我要写书评

暂无评论

A CNN-LSTM Network for Classification of Attention Deficit H...

第41届中国控制会议

作者： Yaru Zhang Xiangxin Tan Yue Gu Key Laboratory of Computer Vision and System (Ministry of Education) School of Computer Science and EngineeringTianjin University of Technology Engineering Research Center of Learning-Based Intelligent system Ministry of Education

ISBN: (数字)9789887581536

ISBN: (纸本)9781665482561

Attention deficit hyperactivity disorder(ADHD) is one of the most common neuropsychiatric disorders in children,with core symptoms of inattention,impulsivity,and *** a result,early detection and treatment are *** this point,ADHD can be diagnosed using an electroencephalogram(EEG).This study proposes a model for identifying the EEG of children with *** proposed model is made up of three parts:temporal convolutional blocks,spatial convolutional blocks,and Long Short Term Memory(LSTM),and because EEG contains more temporal information than other data,we build a model that can extract more temporal *** study included 24 subjects,including 12 children with ADHD and12 normal *** experimental results show that our proposed model outperforms Support Vector Machine(SVM) and EEGNet,with our model achieving correct classification rates of 92.29%,92.76%,and 90.91% on the three backs,respectively,and 94.25% on the full data,demonstrating the model's feasibility and applicability to the ADHD recognition problem.

关键词： Attention Deficit Hyperactivity Disorder Electroencephalogram Deep learning Classification

GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Cepeda, Vicente Vivanco Nayak, Gaurav Kumar Shah, Mubarak Center for Research in Computer Vision University of Central Florida United States

Worldwide Geo-localization aims to pinpoint the precise location of images taken anywhere on Earth. This task has considerable challenges due to the immense variation in geographic landscapes. The image-to-image retrieval-based approaches fail to solve this problem on a global scale as it is not feasible to construct a large gallery of images covering the entire world. Instead, existing approaches divide the globe into discrete geographic cells, transforming the problem into a classification task. However, their performance is limited by the predefined classes and often results in inaccurate localizations when an image’s location significantly deviates from its class center. To overcome these limitations, we propose GeoCLIP, a novel CLIP-inspired Image-to-GPS retrieval approach that enforces alignment between the image and its corresponding GPS locations. GeoCLIP’s location encoder models the Earth as a continuous function by employing positional encoding through random Fourier features and constructing a hierarchical representation that captures information at varying resolutions to yield a semantically rich high-dimensional feature suitable to use even beyond geo-localization. To the best of our knowledge, this is the first work employing GPS encoding for geo-localization. We demonstrate the efficacy of our method via extensive experiments and ablations on benchmark datasets. We achieve competitive performance with just 20% of training data, highlighting its effectiveness even in limited-data settings. Furthermore, we qualitatively demonstrate geo-localization using a text query by leveraging the CLIP backbone of our image encoder. The project webpage is available at: https://***/GeoCLIP Copyright © 2023, The Authors. All rights reserved.

关键词： Location

AlignDet: Aligning Pre-training and Fine-tuning in Object Detection

学校读者我要写书评

暂无评论

AlignDet: Aligning Pre-training and Fine-tuning in Object De...

International Conference on computer vision (ICCV)

作者： Ming Li Jie Wu Xionghui Wang Chen Chen Jie Qin Xuefeng Xiao Rui Wang Min Zheng Xin Pan ByteDance Inc Center for Research in Computer Vision University of Central Florida

The paradigm of large-scale pre-training followed by downstream fine-tuning has been widely employed in various object detection algorithms. In this paper, we reveal discrepancies in data, model, and task between the pre-training and fine-tuning procedure in existing practices, which implicitly limit the detector’s performance, generalization ability, and convergence speed. To this end, we propose AlignDet, a unified pre-training framework that can be adapted to various existing detectors to alleviate the discrepancies. AlignDet decouples the pre-training process into two stages, i.e., image-domain and box-domain pre-training. The image-domain pre-training optimizes the detection backbone to capture holistic visual abstraction, and box-domain pre-training learns instance-level semantics and task-aware concepts to initialize the parts out of the backbone. By incorporating the self-supervised pretrained backbones, we can pre-train all modules for various detectors in an unsupervised paradigm. As depicted in Figure 1, extensive experiments demonstrate that AlignDet can achieve significant improvements across diverse protocols, such as ${\color{Green}\text{detection algorithms}}, {\color{Blue}\text{model backbones}}, {\color{Red}\text{data settings}}$, and ${\color{SkyBlue}\text{training schedules}}$. For example, AlignDet improves FCOS by 5.3 mAP, RetinaNet by 2.1 mAP, Faster R-CNN by 3.3 mAP, and DETR by 2.3 mAP under fewer epochs.

关键词：

Egocentric RGB+Depth Action Recognition in Industry-Like Settings

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Kini, Jyoti Fleischer, Sarah Dave, Ishan Shah, Mubarak Center for Research in Computer Vision University of Central Florida United States

Action recognition from an egocentric viewpoint is a crucial perception task in robotics and enables a wide range of human-robot interactions. While most computer vision approaches prioritize the RGB camera, the Depth modality—which can further amplify the subtleties of actions from an egocentric perspective—remains underexplored. Our work focuses on recognizing actions from egocentric RGB and Depth modalities in an industry-like environment. To study this problem, we consider the recent MECCANO dataset, which provides a wide range of assembling actions. Our framework is based on the 3D Video SWIN Transformer to encode both RGB and Depth modalities effectively. To address the inherent skewness in real-world multimodal action occurrences, we propose a training strategy using an exponentially decaying variant of the focal loss modulating factor. Additionally, to leverage the information in both RGB and Depth modalities, we opt for late fusion to combine the predictions from each modality. We thoroughly evaluate our method on the action recognition task of the MECCANO dataset, and it significantly outperforms the prior work. Notably, our method also secured first place at the multimodal action recognition challenge at ICIAP 2023. © 2023, CC BY-NC-SA.

关键词： Human robot interaction

3D facial attractiveness prediction based on deep feature fusion

学校读者我要写书评

暂无评论

3D facial attractiveness prediction based on deep feature fu...

作者： Liu, Yu Huang, Enquan Zhou, Ziyu Wang, Kexuan Liu, Shu College of Systems Engineering National University of Defense Technology Changsha China School of Computer Science and Engineering Central South University Changsha China Hunan Engineering Research Center of Machine Vision and Intelligent Medicine Changsha China

Facial attractiveness prediction is an important research topic in the computer vision community. It not only contributes to the development of interdisciplinary research in psychology and sociology, but also provides fundamental technical support for applications like aesthetic medicine and social media. With the advances in 3D data acquisition and feature representation, this paper aims to investigate the facial attractiveness from deep learning and three-dimensional perspectives. The 3D faces are first processed to unwrap the texture images and refine the raw meshes. The feature extraction networks for texture, point cloud, and mesh are then delicately designed, considering the characteristics of different types of data. A more discriminative face representation is derived by feature fusion for the final attractiveness prediction. During network training, the cyclical learning rate with an improved range test is introduced, so as to alleviate the difficulty in hyperparameter setting. Extensive experiments are conducted on a 3D FAP benchmark, where the results demonstrate the significance of deep feature fusion and enhanced learning rate in cooperatively facilitating the performance. Specifically, the fusion of texture image and point cloud achieves the best overall prediction, with PC, MAE, and RMSE of 0.7908, 0.4153, and 0.5231, respectively. © 2023 John Wiley & Sons Ltd.

关键词： Forecasting

TeD-SPAD: Temporal Distinctiveness for Self-supervised Privacy-preservation for video Anomaly Detection

学校读者我要写书评

暂无评论

TeD-SPAD: Temporal Distinctiveness for Self-supervised Priva...

International Conference on computer vision (ICCV)

作者： Joseph Fioresi Ishan Rajendrakumar Dave Mubarak Shah Center for Research in Computer Vision University of Central Florida Orlando USA

Video anomaly detection (VAD) without human monitoring is a complex computer vision task that can have a positive impact on society if implemented successfully. While recent advances have made significant progress in solving this task, most existing approaches overlook a critical real-world concern: privacy. With the increasing popularity of artificial intelligence technologies, it becomes crucial to implement proper AI ethics into their development. Privacy leakage in VAD allows models to pick up and amplify unnecessary biases related to people’s personal information, which may lead to undesirable decision making. In this paper, we propose TeD-SPAD, a privacy-aware video anomaly detection framework that destroys visual private information in a self-supervised manner. In particular, we propose the use of a temporally-distinct triplet loss to promote temporally discriminative features, which complements current weakly-supervised VAD methods. Using TeD-SPAD, we achieve a positive trade-off between privacy protection and utility anomaly detection performance on three popular weakly supervised VAD datasets: UCF-Crime, XD-Violence, and ShanghaiTech. Our proposed anonymization model reduces private attribute prediction by 32.25% while only reducing frame-level ROC AUC on the UCF-Crime anomaly detection dataset by 3.69%.

关键词：

BPCoach: Exploring Hero Drafting in Professional MOBA Tournaments via Visual Analytics

学校读者我要写书评

暂无评论

Proceedings of the ACM on Human-computer Interaction 2024年第CSCW1期8卷 1-31页

作者： Liu, Shiyi Ma, Ruofei Zhao, Chuyi Li, Zhenbang Xiao, Jianpeng Li, Quan School of Information Science and Technology ShanghaiTech University Shanghai China Department of Computer Science Tufts University MedfordMA United States Guangzhou Qucheng Culture Media Co. Guangzhou China School of Information Science and Technology ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging Shanghai China

Hero drafting for multiplayer online arena (MOBA) games is crucial because drafting directly affects the outcome of a match. Both sides take turns to "ban"/"pick" a hero from a roster of approximately 100 heroes to assemble their drafting. In professional tournaments, the process becomes more complex as teams are not allowed to pick heroes used in the previous rounds with the "best-of-N" rule. Additionally, human factors including the team's familiarity with drafting and play styles are overlooked by previous studies. Meanwhile, the huge impact of patch iteration on drafting strengths in the professional tournament is of concern. To this end, we propose a visual analytics system, BPCoach, to facilitate hero drafting planning by comparing various drafting through recommendations and predictions and distilling relevant human and in-game factors. Two case studies, expert feedback, and a user study suggest that BPCoach helps determine hero drafting in a rounded and efficient manner. © 2023 Copyright held by the owner/author(s).

关键词： Visualization

LucidDreaming: Controllable Object-Centric 3D Generation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wang, Zhaoning Li, Ming Chen, Chen Center for Research in Computer Vision University of Central Florida United States

With the recent development of generative models, Text-to-3D generations have also seen significant growth, opening a door for creating video-game 3D assets from a more general public. Nonetheless, people without any professional 3D editing experience would find it hard to achieve precise control over the 3D generation, especially if there are multiple objects in the prompt, as using text to control often leads to missing objects and imprecise locations. In this paper, we present LucidDreaming as an effective pipeline capable of spatial and numerical control over 3D generation from only textual prompt commands or 3D bounding boxes. Specifically, our research demonstrates that Large Language Models (LLMs) possess 3D spatial awareness and can effectively translate textual 3D information into precise 3D bounding boxes. We leverage LLMs to get individual object information and their 3D bounding boxes as the initial step of our process. Then with the bounding boxes, We further propose clipped ray sampling and object-centric density blob bias to generate 3D objects aligning with the bounding boxes. We show that our method exhibits remarkable adaptability across a spectrum of mainstream Score Distillation Sampling-based 3D generation frameworks and our pipeline can even used to insert objects into an existing NeRF scene. Moreover, we also provide a dataset of prompts with 3D bounding boxes, benchmarking 3D spatial controllability. With extensive qualitative and quantitative experiments, we demonstrate that LucidDreaming achieves superior results in object placement precision and generation fidelity compared to current approaches, while maintaining flexibility and ease of use for non-expert users. Videos are available at: https://***/LucidDreaming/. Copyright © 2023, The Authors. All rights reserved.

关键词： 3D modeling