检索结果-内蒙古大学图书馆

Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Jiao, Yang Chen, Shaoxiang Jie, Zequn Chen, Jingjing Ma, Lin Jiang, Yu-Gang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center on Intelligent Visual Computing China Meituan

Large Multimodal Model (LMM) is a hot research topic in the computer vision area and has also demonstrated remarkable potential across multiple disciplinary fields. A recent trend is to further extend and enhance the perception capabilities of LMMs. The current methods follow the paradigm of adapting the visual task outputs to language-oriented formats. This adaptation leads to the convenient development of such LMMs with minimal modifications, however, it overlooks the intrinsic characteristics of diverse visual tasks and hinders the learning of perception capabilities. To address this issue, we propose a novel LMM architecture named Lumen, which decouples the learning of perception capabilities into task-agnostic and task-specific stages. Firstly, Lumen promotes fine-grained vision-language concept alignment, which is the fundamental capability for various visual tasks. Thus the output of the task-agnostic stage is a shared representation for all vision-centric tasks we address in this paper. Afterward, the task-specific decoding is carried out by flexibly routing the shared representation to lightweight task decoders with negligible training efforts. Comprehensive experimental results on a series of vision-centric and VQA benchmarks indicate that our Lumen model not only achieves or surpasses the performance of existing LMM-based approaches in a range of vision-centric tasks while maintaining general visual understanding and instruction following capabilities. © 2024, CC BY-NC-SA.

关键词： Decoding

Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Gong, Chao Chen, Kai Wei, Zhipeng Chen, Jingjing Jiang, Yu-Gang Shanghai Key Lab of Intell. Info. Processing School of Computer Science Fudan University China Shanghai Collaborative Innovation Center on Intelligent Visual Computing China

Text-to-image models encounter safety issues, including concerns related to copyright and Not-Safe-For-Work (NSFW) content. Despite several methods have been proposed for erasing inappropriate concepts from diffusion models, they often exhibit incomplete erasure, consume a lot of computing resources, and inadvertently damage generation ability. In this work, we introduce Reliable and Efficient Concept Erasure (RECE), a novel approach that modifies the model in 3 seconds without necessitating additional fine-tuning. Specifically, RECE efficiently leverages a closed-form solution to derive new target embeddings, which are capable of regenerating erased concepts within the unlearned model. To mitigate inappropriate content potentially represented by derived embeddings, RECE further aligns them with harmless concepts in cross-attention layers. The derivation and erasure of new representation embeddings are conducted iteratively to achieve a thorough erasure of inappropriate concepts. Besides, to preserve the model’s generation ability, RECE introduces an additional regularization term during the derivation process, resulting in minimizing the impact on unrelated concepts during the erasure process. All the processes above are in closed-form, guaranteeing extremely efficient erasure in only 3 seconds. Benchmarking against previous approaches, our method achieves more efficient and thorough erasure with minor damage to original generation ability and demonstrates enhanced robustness against red-teaming tools. Code is available at https://***/CharlesGong12/RECE. Copyright © 2024, The Authors. All rights reserved.

关键词： Benchmarking

AIM 2024 Challenge on UHD Blind Photo Quality Assessment

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hosu, Vlad Conde, Marcos V. Agnolucci, Lorenzo Barman, Nabajeet Zadtootaghaj, Saman Timofte, Radu Sun, Wei Zhang, Weixia Cao, Yuqin Cao, Linhan Jia, Jun Chen, Zijian Zhang, Zicheng Min, Xiongkuo Zhai, Guangtao Tan, Songbai Zhang, Lixin Yue, Guanghui Kwon, Daekyu Kim, Dongyoung Kim, Seon Joo Zhang, Yunchen Xu, Xiangkai Gao, Hong Bao, Yiming Shi, Ji Dong, Xiugang Zhou, Xiangsheng Tu, Yaofeng Chen, Zewen Xu, Shunhan Guo, Haochen Zeng, Yun Liu, Shuai Guo, Jian Wang, Juan Li, Bing Liu, Dehua Liu, Hesong Malivenko, Grigory Gerek, Asile Ma, Xingyuan Li, Cheng Lee, Joonhee Bang, Junseo Chun, Se Young Computer Vision Lab CAIDAS IFI University of Würzburg Germany Visual Computing Group FTG Sony PlayStation Sony AI University of Florence Italy Challenge Organizers

We introduce the AIM 2024 UHD-IQA Challenge, a competition to advance the No-Reference Image Quality Assessment (NR-IQA) task for modern, high-resolution photos. The challenge is based on the recently released UHD-IQA Benchmark Database, which comprises 6,073 UHD-1 (4K) images annotated with perceptual quality ratings from expert raters. Unlike previous NR-IQA datasets, UHD-IQA focuses on highly aesthetic photos of superior technical quality, reflecting the ever-increasing standards of digital photography. This challenge aims to develop efficient and effective NR-IQA models. Participants are tasked with creating novel architectures and training strategies to achieve high predictive performance on UHD-1 images within a computational budget of 50G MACs. This enables model deployment on edge devices and scalable processing of extensive image collections. Winners are determined based on a combination of performance metrics, including correlation measures (SRCC, PLCC, KRCC), absolute error metrics (MAE, RMSE), and computational efficiency (G MACs). To excel in this challenge, participants leverage techniques like knowledge distillation, low-precision inference, and multi-scale training. By pushing the boundaries of NR-IQA for high-resolution photos, the UHD-IQA Challenge aims to stimulate the development of practical models that can keep pace with the rapidly evolving landscape of digital photography. The innovative solutions emerging from this competition will have implications for various applications, from photo curation and enhancement to image compression. © 2024, CC BY.

关键词： Budget control

AN INTEGRATED FRAMEWORK FOR DEVELOPING AND EVALUATING AN AUTOMATED LECTURE STYLE ASSESSMENT SYSTEM

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Dimitriadou, Eleni Lanitis, Andreas Visual Media Computing Lab Department of Multimedia and Graphic Arts Cyprus University of Technology Limassol Cyprus CYENS Centre of Excellence Nicosia Cyprus

The aim of the work presented in this paper is to develop and evaluate an integrated system that provides automated lecture style evaluation, allowing teachers to get instant feedback related to the goodness of their lecturing style. The proposed system aims to promote improvement of lecture quality, that could upgrade the overall student learning experience. The proposed application utilizes specific measurable biometric characteristics, such as facial expressions, body activity, speech rate and intonation, hand movement, and facial pose, extracted from a video showing the lecturer from the audience point of view. Measurable biometric features extracted during a lecture are combined to provide teachers with a score reflecting lecture style quality both at frame rate and by providing lecture quality metrics for the whole lecture. The acceptance of the proposed lecture style evaluation system was evaluated by chief education officers, teachers and students regarding the functionality, usefulness of the application, and possible improvements. The results indicate that participants found the application novel and useful in providing automated feedback regarding lecture quality. Furthermore, the performance evaluation of the proposed system was compared with the performance of humans in the task of lecture style evaluation. Results indicate that the proposed system not only achieves similar performance to human observers, but in some cases, it outperforms them. Copyright © 2023, The Authors. All rights reserved.

关键词： Biometrics

SimDA: Simple Diffusion Adapter for Efficient Video Generation

学校读者我要写书评

暂无评论

SimDA: Simple Diffusion Adapter for Efficient Video Generati...

Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Zhen Xing Qi Dai Han Hu Zuxuan Wu Yu-Gang Jiang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing Microsoft Research Asia

ISBN: (数字)9798350353006

ISBN: (纸本)9798350353013

The recent wave of AI-generated content has witnessed the great development and success of Text-to-Image (T2I) technologies. By contrast, Text-to- Video (T2V) still falls short of expectations though attracting increasing interest. Existing works either train from scratch or adapt large T2I model to videos, both of which are computation and re-source expensive. In this work, we propose a Simple Dif-fusion Adapter (SimDA) that fine-tunes only 24M out of *** parameters of a strong T2I model, adapting it to video generation in a parameter-efficient way. In particular, we turn the T2I model for T2V by designing light-weight spatial and temporal adapters for transfer learning. Besides, we change the original spatial attention to the proposed Latent-Shift Attention (LSA) for temporal consistency. With a similar model architecture, we further train a video super-resolution model to generate high-definition (1024 x 1024) videos. In addition to T2V generation in the wild, SimDA could also be utilized in one-shot video editing with only 2 minutes tuning. Doing so, our method could minimize the training effort with extremely few tunable parameters for model adaptation.

关键词： Training Adaptation models Computational modeling Transfer learning Superresolution Text to image Diffusion models

Downstream Task-agnostic Transferable Attacks on Language-Image Pre-training Models

学校读者我要写书评

暂无评论

Downstream Task-agnostic Transferable Attacks on Language-Im...

IEEE International Conference on Multimedia and Expo (ICME)

作者： Yiqiang Lv Jingjing Chen Zhipeng Wei Kai Chen Zuxuan Wu Yu-Gang Jiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing

Vision-language pre-trained models (e.g., CLIP) trained on large-scale datasets via self-supervised learning, are drawing increasing research attention since they can achieve superior performances on multi-modal downstream tasks. Nevertheless, we find that the adversarial perturbations crafted on vision-language pre-trained models can be used to attack different corresponding downstream task models. Specifically, to investigate such adversarial transferability, we introduce a task-agnostic method named Global and Local Augmentation (GLA) attack to generate highly transferable adversarial examples on CLIP, to attack black-box downstream task models. GLA adopts random crop and resize at both global and local patch levels, to create more diversity and make adversarial noises robust. Then GLA generates the adversarial perturbations by minimizing the cosine similarity between intermediate features from augmented adversarial and benign examples. Extensive experiments on three CLIP image encoders with different backbones and three different downstream tasks demonstrate the superiority of our method compared with other strong baselines. The code is available at https://***/yqlvcoding/GLAattack.

关键词：

DirectVoxGO++: Fast Neural Radiance Fields for Object Reconstruction 35

学校读者我要写书评

暂无评论

DirectVoxGO++: Fast Neural Radiance Fields for Object Recons...

35th Conference on Graphics, Patterns, and Images, SIBGRAPI 2022

作者： Perazzo, Daniel Lima, Joao Paulo Velho, Luiz Teichrieb, Veronica Voxar Labs CIn Ufpe Pernambuco Recife Brazil Visual Computing Lab Dc Ufrpe Voxar Labs Pernambuco Recife Brazil Visgraf Impa Impa Rio de Janeiro Rio de Janeiro Brazil

ISBN: (数字)9781665453851

ISBN: (纸本)9781665453851

In recent years, new reconstruction techniques based on Neural Radiance Fields (NeRFs) have created new forms to model objects instead of the traditional mesh and point cloud-based representations, allowing for more photorealistic rendering. However, these techniques were too slow to be used in practical settings, taking in the range of hours in high-end GPUs. Due to these limitations, new techniques have been created for fast reconstruction of scenes, such as DirectVoxGO. Alongside this limitation, one issue with NeRFs is that they were initially unable to separate the foreground from the background and had problems with 360 scenes until the emergence of new techniques such as NeRF++. Our method extends DirectVoxGO to allow the handling of unbounded scenes inspired by some ideas from NeRF++, adapting it to incorporate elements from a neural hashing approach employed by other works. Our technique improved photorealism compared with DirectVoxGO and Plenoxels on a subset of the LF dataset on average in at least 2%, 8%, and 8% for PSNR, SSIM, and LPIPS metrics, respectively, while also being an order of magnitude faster than NeRF++. Code will be available in https://***/danperazzo/dvgoplusplus. © 2022 IEEE.

关键词： Program processors

Detect and Approach: Close-Range Navigation Support for People with Blindness and Low Vision 17th

学校读者我要写书评

暂无评论

Detect and Approach: Close-Range Navigation Support for Pe...

17th European Conference on Computer Vision, ECCV 2022

作者： Hao, Yu Feng, Junchi Rizzo, John-Ross Wang, Yao Fang, Yi NYU Multimedia and Visual Computing Lab New York United States NYU Tandon School of Engineering New York University New York United States New York University Abu Dhabi Abu Dhabi United Arab Emirates NYU Langone Health New York United States

ISBN: (纸本)9783031250743

People with blindness and low vision (pBLV) experience significant challenges when locating final destinations or targeting specific objects in unfamiliar environments. Furthermore, besides initially locating and orienting oneself to a target object, approaching the final target from one’s present position is often frustrating and challenging, especially when one drifts away from the initial planned path to avoid obstacles. In this paper, we develop a novel wearable navigation solution to provide real-time guidance for a user to approach a target object of interest efficiently and effectively in unfamiliar environments. Our system contains two key visual computing functions: initial target object localization in 3D and continuous estimation of the user’s trajectory, both based on the 2D video captured by a low-cost monocular camera mounted on in front of the chest of the user. These functions enable the system to suggest an initial navigation path, continuously update the path as the user moves, and offer timely recommendation about the correction of the user’s path. Our experiments demonstrate that our system is able to operate with an error of less than 0.5 m both outdoor and indoor. The system is entirely vision-based and does not need other sensors for navigation, and the computation can be run with the Jetson processor in the wearable system to facilitate real-time navigation assistance. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

关键词： Eye protection

FairCLIP: Harnessing Fairness in Vision-Language Learning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Luo, Yan Shi, Min Khan, Muhammad Osama Afzal, Muhammad Muneeb Huang, Hao Yuan, Shuaihang Tian, Yu Song, Luo Kouhana, Ava Elze, Tobias Fang, Yi Wang, Mengyu Harvard Ophthalmology AI Lab Harvard University United States Tandon School of Engineering New York University United States Multimedia and Visual Computing Lab New York University Abu Dhabi United States

Fairness is a critical concern in deep learning, especially in healthcare, where these models influence diagnoses and treatment decisions. Although fairness has been investigated in the vision-only domain, the fairness of medical vision-language (VL) models remains unexplored due to the scarcity of medical VL datasets for studying fairness. To bridge this research gap, we introduce the first fair vision-language medical dataset (Harvard-FairVLMed) that provides detailed demographic attributes, ground-truth labels, and clinical notes to facilitate an in-depth examination of fairness within VL foundation models. Using Harvard-FairVLMed, we conduct a comprehensive fairness analysis of two widely-used VL models (CLIP and BLIP2), pre-trained on both natural and medical domains, across four different protected attributes. Our results highlight significant biases in all VL models, with Asian, Male, Non-Hispanic, and Spanish being the preferred subgroups across the protected attributes of race, gender, ethnicity, and language, respectively. In order to alleviate these biases, we propose FairCLIP, an optimal-transport-based approach that achieves a favorable trade-off between performance and fairness by reducing the Sinkhorn distance between the overall sample distribution and the distributions corresponding to each demographic group. As the first VL dataset of its kind, Harvard-FairVLMed holds the potential to catalyze advancements in the development of machine learning models that are both ethically aware and clinically effective. Our dataset and code are available at https://***/datasets/ harvard-fairvlmed10k. © 2024, CC BY-NC-ND.

关键词： Population statistics