检索结果-内蒙古大学图书馆

CLIP-Flow:Decoding images encoded in CLIP space

Computational visual Media 2024年第6期10卷 1157-1168页

作者： Hao Ma Ming Li Jingyuan Yang Or Patashnik Dani Lischinski Daniel Cohen-Or Hui Huang Visual Computing Research Center College of Computer Science and Software EngineeringShenzhen UniversityShenzhen 518060China Department of Computer Science Tel Aviv UniversityTel Aviv 6997801Israel School of Computer Science and Engineering the Hebrew University of JerusalemJerusalem 91904Israel

This study introduces CLIP-Flow,a novel network for generating images from a given image or *** effectively utilize the rich semantics contained in both modalities,we designed a semantics-guided methodology for image-and text-to-image *** particular,we adopted Contrastive Language-Image Pretraining(CLIP)as an encoder to extract semantics and StyleGAN as a decoder to generate images from such ***,to bridge the embedding space of CLIP and latent space of StyleGAN,real NVP is employed and modified with activation normalization and invertible *** the images and text in CLIP share the same representation space,text prompts can be fed directly into CLIP-Flow to achieve text-to-image *** conducted extensive experiments on several datasets to validate the effectiveness of the proposed image-to-image synthesis *** addition,we tested on the public dataset Multi-Modal CelebA-HQ,for text-to-image *** validated that our approach can generate high-quality text-matching images,and is comparable with state-of-the-art methods,both qualitatively and quantitatively.

关键词： image-to-image text-to-image contrastive language-image pretraining(CLIP) flow StyleGAN

来源：评论

学校读者我要写书评

暂无评论

Taming diffusion model for exemplar-based image translation

引用

Computational visual Media 2024年第6期10卷 1031-1043页

作者： Ma, Hao Yang, Jingyuan Huang, Hui Shenzhen University Visual Computing Research Center College of Computer Science and Software Engineering Shenzhen China (GRID:grid.263488.3) (ISNI:0000 0001 0472 9649)

Exemplar-based image translation involves converting semantic masks into photorealistic images that adopt the style of a given ***,most existing GAN-based translation methods fail to produce photorealistic *** this study,we propose a new diffusion model-based approach for generating high-quality images that are semantically aligned with the input mask and resemble an exemplar in *** proposed method trains a conditional denoising diffusion probabilistic model(DDPM)with a SPADE module to integrate the semantic *** then used a novel contextual loss and auxiliary color loss to guide the optimization process,resulting in images that were visually pleasing and semantically *** demonstrate that our method outperforms state-of-the-art approaches in terms of both visual quality and quantitative metrics.

关键词： exemplar image translation denoising diffusion probabilistic model(DDPM)

来源：评论

学校读者我要写书评

暂无评论

Self-Supervised Color-Concept Association via Image Colorization

引用

IEEE Transactions on visualization and computer Graphics 2023年第1期29卷 247-256页

作者： Hu, Ruizhen Ye, Ziqi Chen, Bin Van Kaick, Oliver Huang, Hui Shenzhen University Visual Computing Research Center China Carleton University School of Computer Science Canada

The interpretation of colors in visualizations is facilitated when the assignments between colors and concepts in the visualizations match human's expectations, implying that the colors can be interpreted in a semantic manner. However, manually creating a dataset of suitable associations between colors and concepts for use in visualizations is costly, as such associations would have to be collected from humans for a large variety of concepts. To address the challenge of collecting this data, we introduce a method to extract color-concept associations automatically from a set of concept images. While the state-of-the-art method extracts associations from data with supervised learning, we developed a self-supervised method based on colorization that does not require the preparation of ground truth color-concept associations. Our key insight is that a set of images of a concept should be sufficient for learning color-concept associations, since humans also learn to associate colors to concepts mainly from past visual input. Thus, we propose to use an automatic colorization method to extract statistical models of the color-concept associations that appear in concept images. Specifically, we take a colorization model pre-trained on ImageNet and fine-tune it on the set of images associated with a given concept, to predict pixel-wise probability distributions in Lab color space for the images. Then, we convert the predicted probability distributions into color ratings for a given color library and aggregate them for all the images of a concept to obtain the final color-concept associations. We evaluate our method using four different evaluation metrics and via a user study. Experiments show that, although the state-of-the-art method based on supervised learning with user-provided ratings is more effective at capturing relative associations, our self-supervised method obtains overall better results according to metrics like Earth Mover's Distance (EMD) and Entropy Differenc

关键词： Data mining

来源：评论

学校读者我要写书评

暂无评论

Making Large Language Models Better Reasoners with Orchestrated Streaming Experiences

Making Large Language Models Better Reasoners with Orchestra...

引用

2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024

作者： Liu, Xiangyang He, Junliang Qiu, Xipeng School of Computer Science Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing China

ISBN: (纸本)9798891761643

Large language models (LLMs) can perform complex reasoning by generating intermediate thoughts under zero-shot or few-shot settings. However, zero-shot prompting always encounters low performance, and the superior performance of few-shot prompting hinges on the manual-crafted demonstrations. In this paper, we present RoSE (Reasoning with Orchestrated Streaming Experiences), a general framework for solving reasoning tasks that can self-improve without complex external efforts. To enable RoSE, we describe an architecture that extends an LLM to store all answered questions and their thoughts in a streaming experience pool then orchestrates helpful questions from the pool to assist in answering new questions. To set up a question-aware orchestration mechanism, RoSE first calculates the similarity of each question in the pool with a new test question. Since the solution to each answered question is not always correct, RoSE will sort the questions according to their similarity with the new question, and then uniformly divide them into multiple buckets. It finally extracts one question from each bucket to make these extracted questions more diverse. To make these extracted questions help RoSE answer new questions as much as possible, we introduce two other attributes of uncertainty and complexity for each question. RoSE will preferentially select the questions with low uncertainty and high complexity from each bucket. We evaluate the versatility of RoSE in various reasoning tasks, LLMs, and CoT methods. © 2024 Association for Computational Linguistics.

关键词： Computational linguistics

来源：评论

学校读者我要写书评

暂无评论

Trace-based Multi-Dimensional Root Cause Localization of Performance Issues in Microservice Systems 24

Trace-based Multi-Dimensional Root Cause Localization of Per...

引用

44th ACM/IEEE International Conference on Software Engineering, ICSE 2024

作者： Zhang, Chenxi Dong, Zhen Peng, Xin Zhang, Bicheng Chen, Miao Fudan University China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China School of Computer Science and Shanghai Key Laboratory of Data Science Fudan University China

ISBN: (纸本)9798400702174

Modern microservice systems have become increasingly complicated due to the dynamic and complex interactions and runtime environment. It leads to the system vulnerable to performance issues caused by a variety of reasons, such as the runtime environments, communications, coordinations, or implementations of services. Traces record the detailed execution process of a request through the system and have been widely used in performance issues diagnosis in microservice systems. By identifying the execution processes and attribute value combinations that are common in anomalous traces but rare in normal traces, engineers may localize the root cause of a performance issue into a smaller scope. However, due to the complex structure of traces and the large number of attribute combinations, it is challenging to find the root cause from the huge search space. In this paper, we propose TraceContrast, a trace-based multidimensional root cause localization approach. TraceContrast uses a sequence representation to describe the complex structure of a trace with attributes of each span. Based on the representation, it combines contrast sequential pattern mining and spectrum analysis to localize multidimensional root causes efficiently. Experimental studies on a widely used microservice benchmark show that TraceContrast outperforms existing approaches in both multidimensional and instance-dimensional root cause localization with significant accuracy advantages. Moreover, Trace-Contrast is efficient and its efficiency can be further improved by parallel execution. © 2024 ACM.

关键词： Spectrum analysis

来源：评论

学校读者我要写书评

暂无评论

Mixture of Experts for Audio-visual Learning 38

Mixture of Experts for Audio-Visual Learning

引用

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Cheng, Ying Li, Yang He, Junjie Feng, Rui School of Computer Science Fudan University China Shanghai Key Laboratory of Intelligent Information Processing China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China

With the rapid development of multimedia technology, audio-visual learning has emerged as a promising research topic within the field of multimodal analysis. In this paper, we explore parameter-efficient transfer learning for audio-visual learning and propose the Audio-visual Mixture of Experts (AVMoE) to inject adapters into pre-trained models flexibly. Specifically, we introduce unimodal and cross-modal adapters as multiple experts to specialize in intra-modal and inter-modal information, respectively, and employ a lightweight router to dynamically allocate the weights of each expert according to the specific demands of each task. Extensive experiments demonstrate that our proposed approach AVMoE achieves superior performance across multiple audio-visual tasks, including AVE, AVVP, AVS, and AVQA. Furthermore, visual-only experimental results also indicate that our approach can tackle challenging scenes where modality information is missing. The source code is available at https://***/yingchengy/AVMOE. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Retrieval Augmented Recipe Generation

Retrieval Augmented Recipe Generation

引用

2025 IEEE/CVF Winter Conference on Applications of computer Vision, WACV 2025

作者： Liu, Guoshan Yin, Hailong Zhu, Bin Chen, Jingjing Ngo, Chong-Wah Jiang, Yu-Gang School of Computer Science Fudan University Shanghai Key Lab of Intelligent Information Processing China Shanghai Collaborative Innovation Center on Intelligent Visual Computing China Singapore Management University Singapore

ISBN: (纸本)9798331510831

The growing interest in generating recipes from food images has drawn substantial research attention in recent years. Existing works for recipe generation primarily utilize a two-stage training method - first predicting ingredients from a food image and then generating instructions from both the image and ingredients. Large Multi-modal Models (LMMs), which have achieved notable success across a variety of vision and language tasks, shed light on generating both ingredients and instructions directly from images. Nevertheless, LMMs still face the common issue of hallu-cinations during recipe generation, leading to suboptimal performance. To tackle this issue, we propose a retrieval augmented large multimodal model for recipe generation. We first introduce Stochastic Diversified Retrieval Augmentation (SDRA) to retrieve recipes semantically related to the image from an existing datastore as a supplement, integrating them into the prompt to add diverse and rich context to the input image. Additionally, Self-Consistency Ensemble Voting mechanism is proposed to determine the most confident prediction recipes as the final output. It calculates the consistency among generated recipe candidates, which use different retrieval recipes as context for generation. Extensive experiments validate the effectiveness of our proposed method, which demonstrates state-of-the-art (SOTA) performance in recipe generation on the Recipe1M dataset. © 2025 IEEE.

关键词： Food ingredients

来源：评论

学校读者我要写书评

暂无评论

VIPNet: Combining Viewpoint Information and Shape Priors for Instant Multi-view 3D Reconstruction 17th

VIPNet: Combining Viewpoint Information and Shape Priors fo...

引用

17th Asian Conference on computer Vision, ACCV 2024

作者： Ye, Weining Li, Zhixuan Jiang, Tingting School of Computer Science Peking University Beijing100871 China College of Computing and Data Science Nanyang Technological University Singapore Singapore National Engineering Research Center of Visual Technology National Key Laboratory for Multimedia Information Processing School of Computer Science National Biomedical Imaging Center Peking University Beijing100871 China

ISBN: (纸本)9789819609680

While the multi-view 3D reconstruction task has made significant progress, existing methods simply fuse multi-view image features without effectively leveraging available auxiliary information, especially the viewpoint information for guiding and associating features of different views. To this end, we propose to enhance multi-view 3D reconstruction with the power of viewpoint information. Specifically, a simple-yet-effective viewpoint estimator is designed to learn and provide comprehensive viewpoint knowledge for locating and associating learned features from different views. Moreover, to improve the 3D reconstruction quality when 2D images of only very few viewpoints are available, we propose to learn the shape prior knowledge to provide sufficient shape information for compensating the limited 2D observations. Overall, we present VIPNet, benefiting from Viewpoint Information and Shape Prior learning for high-quality multi-view 3D reconstruction. Extensive experiments validate the effectiveness of the proposed VIPNet, which achieves state-of-the-art performance on challenging datasets and shows well generalization ability in real-world scenarios. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.

关键词： 3D modeling

来源：评论

学校读者我要写书评

暂无评论

SCTAP: Supporting Scenario-Centric Trigger-Action Programming based on Software-Defined Physical Environments 23

SCTAP: Supporting Scenario-Centric Trigger-Action Programmin...

引用

2023 World Wide Web Conference, WWW 2023

作者： Sun, Bingkun Shen, Liwei Peng, Xin Wang, Ziming Fudan University China School of Computer Science Shanghai Key Laboratory of Data Science Fudan University China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China

ISBN: (纸本)9781450394161

The physical world we live in is accelerating digitalization with the vigorous development of Internet of Things (IoT). Following this trend, Web of Things (WoT) further enables fast and efficient creation of various applications that perceive and act on the physical world using standard Web technologies. A popular way for creating WoT applications is Trigger-Action Programming (TAP), which allows users to orchestrate the capabilities of IoT devices in the form of "if trigger, then action". However, existing TAP approaches don't support scenario-centric WoT applications which involve abstract modeling of physical environments and complex spatio-temporal dependencies between events and actions. In this paper, we propose an approach called SCTAP which supports Scenario-Centric Trigger-Action Programming based on software-defined physical environments. SCTAP defines a structured and conceptual representation for physical environments, which provides the required programming abstractions for WoT applications. Based on the representation, SCTAP defines a grammar for specifying scenario-centric WoT applications with spatio-temporal dependencies. Furthermore, we design a service-based architecture for SCTAP which supports the integration of device access, event perception, environment representation, and rule execution in a loosely-coupled and extensible way. We implement SCTAP as a WoT infrastructure and evaluate it with two case studies including a smart laboratory and a smart coffee house. The results confirm the usability, feasibility and efficiency of SCTAP and its implementation. © 2023 ACM.

关键词： Internet of things

来源：评论

学校读者我要写书评

暂无评论

TPS++: Attention-Enhanced Thin-Plate Spline for Scene Text Recognition 32

TPS++: Attention-Enhanced Thin-Plate Spline for Scene Text R...

引用

32nd International Joint Conference on Artificial Intelligence, IJCAI 2023

作者： Zheng, Tianlun Chen, Zhineng Bai, Jinfeng Xie, Hongtao Jiang, Yu-Gang Shanghai Collaborative Innovation Center of Intelligent Visual Computing School of Computer Science Fudan University China Tomorrow Advance Life China University of Science and Technology of China China

ISBN: (纸本)9781956792034

Text irregularities pose significant challenges to scene text recognizers. Thin-Plate Spline (TPS)based rectification is widely regarded as an effective means to deal with them. Currently, the calculation of TPS transformation parameters purely depends on the quality of regressed text borders. It ignores the text content and often leads to unsatisfactory rectified results for severely distorted text. In this work, we introduce TPS++, an attention-enhanced TPS transformation that incorporates the attention mechanism to text rectification for the first time. TPS++ formulates the parameter calculation as a joint process of foreground control point regression and content-based attention score estimation, which is computed by a dedicated designed gated-attention block. TPS++ builds a more flexible content-aware rectifier, generating a natural text correction that is easier to read by the subsequent recognizer. Moreover, TPS++ shares the feature backbone with the recognizer in part and implements the rectification at feature-level rather than image-level, incurring only a small overhead in terms of parameters and inference time. Experiments on public benchmarks show that TPS++ consistently improves the recognition and achieves state-of-the-art accuracy. Meanwhile, it generalizes well on different backbones and recognizers. Code is at https://***/simplify23/TPS PP. © 2023 International Joint Conferences on Artificial Intelligence. All rights reserved.

关键词： Character recognition

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：