检索结果-内蒙古大学图书馆

Semantic Object Accuracy for Generative text-to-image synthesis

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022年第3期44卷 1552-1565页

作者： Hinz, Tobias Heinrich, Stefan Wermter, Stefan Univ Hamburg Knowledge Technol Grp D-22527 Hamburg Germany

Generative adversarial networks conditioned on textual image descriptions are capable of generating realistic-looking images. However, current methods still struggle to generate images based on complex image captions from a heterogeneous domain. Furthermore, quantitatively evaluating these text-to-image models is challenging, as most evaluation metrics only judge image quality but not the conformity between the image and its caption. To address these challenges we introduce a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption. The SOA uses a pre-trained object detector to evaluate if a generated image contains objects that are mentioned in the image caption, e.g., whether an image generated from "a car driving down the street" contains a car. We perform a user study comparing several text-to-image models and show that our SOA metric ranks the models the same way as humans, whereas other metrics such as the Inception Score do not. Our evaluation also shows that models which explicitly model objects outperform models which only model global image characteristics.

关键词： Layout Semantics Gallium nitride Measurement Generators image resolution image quality text-to-image synthesis generative adversarial network (GAN) evaluation of generative models generative models

来源：评论

学校读者我要写书评

暂无评论

TRAINING-FREE LOCATION-AWARE text-to-image synthesis 30

TRAINING-FREE LOCATION-AWARE TEXT-TO-IMAGE SYNTHESIS

引用

30th IEEE International Conference on image Processing (ICIP)

作者： Mao, Jiafeng Wang, Xueting Univ Tokyo Dept Informat & Commun Engn Tokyo Japan CyberAgent Inc AI Lab Tokyo Japan

ISBN: (纸本)9781728198354

Current large-scale generative models have impressive efficiency in generating high-quality images based on text prompts. However, they lack the ability to precisely control the size and position of objects in the generated image. In this study(1), we analyze the generative mechanism of the stable diffusion model and propose a new interactive generation paradigm that allows users to specify the position of generated objects without additional training. Moreover, we propose an object detection-based evaluation metric to assess the control capability of location aware generation task. Our experimental results show that our method outperforms state-of-the-art methods on both control capacity and image quality.

关键词： diffusion model text-to-image synthesis

来源：评论

学校读者我要写书评

暂无评论

TDG-Diff: Advancing customized text-to-image synthesis with two-stage diffusion guidance

引用

COMPUTERS & GRAPHICS-UK 2024年 122卷

作者： Lin, Hong Chen, Qi Liu, Chun Hu, Jingsong Wuhan Univ Technol Wuhan 430070 Peoples R China

Recently, there has been widespread attention and significant progress in customized text-to-image synthesis based on diffusion models. However, reconstructing multiple concepts in the same scene remains highly challenging. Therefore, we propose a novel framework called TDG-Diff, which employs a two-stage diffusion guidance to achieve customized image synthesis with multiple concepts. TDG-Diff focuses on improving the sampling process of the diffusion model. Specifically, TDG-Diff subdivides the sampling process into two key stages: attribute separation and appearance refinement, introducing spatial constraints and concept representations for sampling guidance. In the attribute separation stage, TDG-Diff introduces a novel attention modulation method. This method effectively separates the attributes of different concepts based on spatial constraint information, reducing the risk of entanglement between attributes of different concepts. In the appearance refinement stage, TDG-Diff proposes a fusion sampling approach, which combines global text descriptions and concept representations to optimize and enhance the model's ability to capture and represent fine-grained details of concepts. Extensive qualitative and quantitative results demonstrate the effectiveness of TDG-Diff in customized text-to-image synthesis.

关键词： Customized image synthesis Diffusion model Multimodal text-to-image synthesis

来源：评论

学校读者我要写书评

暂无评论

Exploring Progress in text-to-image synthesis: An In-Depth Survey on the Evolution of Generative Adversarial Networks

引用

IEEE ACCESS 2024年 12卷 178401-178440页

作者： Habib, Md Ahsan Wadud, Md Anwar Hussen Patwary, Md Fazlul Karim Rahman, Mohammad Motiur Mridha, M. F. Okuyama, Yuichi Shin, Jungpil Mawlana Bhashani Sci & Technol Univ Dept Comp Sci & Engn Tangail 1902 Bangladesh Sunamgonj Sci & Technol Univ Dept Comp Sci & Engn Sunamganj 3000 Bangladesh Jahangirnagar Univ Inst Informat Technol Dhaka 1342 Bangladesh Amer Int Univ Bangladesh Dept Comp Sci Dhaka 1229 Bangladesh Univ Aizu Sch Comp Sci & Engn Aizu Wakamatsu 9658580 Japan

The emergence of generative adversarial networks (GANs) has ignited substantial interest in the domain of synthesizing images from textual descriptions. This approach has demonstrated remarkable versatility and user-friendliness in producing conditioned images, showcasing notable progress in areas like diversity, visual realism, and semantic alignment in recent years. Notwithstanding these developments, the discipline still faces difficulties, such as producing high-resolution pictures with several objects and developing trustworthy evaluation standards that are in line with human vision. The goal of this study is to provide a comprehensive overview of the state of stochastic text-to-image creation models as of right now. It examines how they have changed over the previous five years and suggests a classification system depending on the degree of supervision required. The paper highlights shortcomings, provides a critical evaluation of current approaches for assessing text-to-image synthesizing models, and suggests further study areas. These goals include improving the training of models and designs for architecture, developing more reliable assessment criteria, and fine-tuning datasets. This review, which focuses on text-to-image synthesizing, is a useful addition to earlier surveys on adversarial networks that are generative and offers guidance for future studies on the subject.

关键词： attention mechanism C-GAN Generative adversarial networks text-to-image synthesis text-to-image synthesis C-GAN text-to-image synthesis

来源：评论

学校读者我要写书评

暂无评论

Language-vision matching for text-to-image synthesis with context-aware GAN

引用

EXPERT SYSTEMS WITH APPLICATIONS 2024年第PartB期255卷

作者： Hou, Yingli Zhang, Wei Zhu, Zhiliang Yu, Hai Northeastern Univ Software Coll Shenyang 110819 Peoples R China Natl Frontiers Sci Ctr Ind Intelligence & Syst Opt Shenyang 110819 Peoples R China Northeastern Univ Key Lab Data Analyt & Optimizat Smart Ind Minist Educ Shenyang Peoples R China

text-to-image generation (T2I) aims to produce visually compelling images while maintaining a high degree of semantic consistency with textual descriptions. Despite the impressive progress made by existing methods, there are problems with limited details of synthesized images and insufficient correlation between the provided text description and the generated images. To address these issues, we propose a Context-Aware Generative Adversarial Network (CA-GAN), which generates images aligned with the input text representations. Specifically, the Context-Aware Block (CA-Block) learns a semantic-adaptive transformation based on text style, enabling the effective fusion of text descriptions and image features for high-quality image generation with better language-vision matching. Furthermore, we propose an Attention Convolution Module (ACM) to identify greater representative traits and avoid the inability to capitalize on non-local contextual information, which enables our model to generate images that exhibit numerous detailed attributes while maintaining highquality and semantic consistency. Thereafter, we integrate self-attention with convolution to enhance feature maps to reinforce the semantic information in the discriminator, emphasizing critical feature channels while suppressing extraneous information, ultimately yielding more detailed and richer images. The experimental results demonstrate the superiority of our method over the SOTA approaches. In addition, further studies confirm the effectiveness of the generated visual details, which exhibit a high degree of alignment with the input text descriptions. Notably, our attention mechanism showcases cooperative effects contributing to overall performance improvement. The code is available at: https://***/hylneu/CAGAN.

关键词： text-to-image synthesis Language-vision matching Multi-modal semantic consistency Attention mechanism

来源：评论

学校读者我要写书评

暂无评论

AtHom: Two Divergent Attentions Stimulated By Homomorphic Training in text-to-image synthesis 22

AtHom: Two Divergent Attentions Stimulated By Homomorphic Tr...

引用

30th ACM International Conference on Multimedia (MM)

作者： Shi, Zhenbo Chen, Zhi Xu, Zhenbo Yang, Wei Huang, Liusheng Univ Sci & Technol China Hefei Peoples R China

ISBN: (纸本)9781450392037

image generation from text is a challenging and ill-posed task. images generated from previous methods usually have low semantic consistency with texts and the achieved resolution is limited. To generate semantically consistent high-resolution images, we propose a novel method named AtHom, in which two attention modules are developed to extract the relationships from both independent modality and unified modality. The first is a novel Independent Modality Attention Module (IAM), which is presented to find out semantically important areas in generated images and to extract the informative context in texts. The second is a new module named Unified Semantic Space Attention Module (UAM), which is utilized to find out the relationships between extracted text context and essential areas in generated images. In particular, to bring the semantic features of texts and images closer in a unified semantic space, AtHom incorporates a homomorphic training mode by exploiting an extra discriminator to distinguish between two different modalities. Extensive experiments show that our AtHom surpasses previous methods by large margins.

关键词： text-to-image synthesis image Generation Attention Generative Adversarial Networks

来源：评论

学校读者我要写书评

暂无评论

TISE: Bag of Metrics for text-to-image synthesis Evaluation 17th

TISE: Bag of Metrics for Text-to-Image Synthesis Evaluation

引用

17th European Conference on Computer Vision (ECCV)

作者： Dinh, Tan M. Rang Nguyen Binh-Son Hua VinAI Res Hanoi Vietnam

ISBN: (纸本)9783031200588;9783031200595

In this paper, we conduct a study on the state-of-the-art methods for text-to-image synthesis and propose a framework to evaluate these methods. We consider syntheses where an image contains a single or multiple objects. Our study outlines several issues in the current evaluation pipeline: (i) for image quality assessment, a commonly used metric, e.g., Inception Score (IS), is often either miscalibrated for the single-object case or misused for the multi-object case;(ii) for text relevance and object accuracy assessment, there is an overfitting phenomenon in the existing R-precision (RP) and Semantic Object Accuracy (SOA) metrics, respectively;(iii) for multi-object case, many vital factors for evaluation, e.g., object fidelity, positional alignment, counting alignment, are largely dismissed;(iv) the ranking of the methods based on current metrics is highly inconsistent with real images. To overcome these issues, we propose a combined set of existing and new metrics to systematically evaluate the methods. For existing metrics, we offer an improved version of IS named IS* by using temperature scaling to calibrate the confidence of the classifier used by IS;we also propose a solution to mitigate the overfitting issues of RP and SOA. For new metrics, we develop counting alignment, positional alignment, object-centric IS, and object-centric FID metrics for evaluating the multi-object case. We show that benchmarking with our bag of metrics results in a highly consistent ranking among existing methods that is well-aligned with human evaluation. As a by-product, we create AttnGAN++, a simple but strong baseline for the benchmark by stabilizing the training of AttnGAN using spectral normalization. We also release our toolbox, so-called TISE, for advocating fair and consistent evaluation of text-to-image models.

关键词： Language and vision Metrics text-to-image synthesis

来源：评论

学校读者我要写书评

暂无评论

AttnGAN: Realistic text-to-image synthesis with Attentional Generative Adversarial Networks 18th

AttnGAN: Realistic Text-to-Image Synthesis with Attentional ...

引用

18th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT)

作者： Mathesul, Shubham Bhutkar, Ganesh Rambhad, Ayush Vishwakarma Inst Technol Pune Maharashtra India

ISBN: (纸本)9783030983888;9783030983871

In this paper, we propose a prototype design for manifold refinement to fine grained text-to-image generation by using Attentional Generative Adversarial Network (AttnGAN) We concentrate on creating realistic images from text descriptions. We have used a collection of Attentional Generative Adversarial Network layers that are able to correctly select the modal meaning at the word-level and sentence-level. Generative Adversarial Networks (GANs) prove to be fundamental structure for many design applications from Game design, Art, Science and Modelling applications. We use GANs for contrastive learning and as a information maximisation approach, and we do extensive research to find the further advancements in image generation. Our prototype is easy to implement and practical;choosing the most relevant word vectors and using those vectors to generate related image sub-regions. The prototype in its current state generates image designs only for the bird species to satisfy the claim for its image generation ability. With due consideration to findings of usability testing, the develpment team in future iterations of the application, hopes to improve the generated image resolution. They plan to provide a choice for created variety of images with further improvements to the image generation algorithm.

关键词： GAN text-to-image synthesis Artificial intelligence Artificial neural networks DAMSM Attentional Generative Adversarial Networks

来源：评论

学校读者我要写书评

暂无评论

Disentangled Representations and Hierarchical Refinement of Multi-Granularity Features for text-to-image synthesis 22

Disentangled Representations and Hierarchical Refinement of ...

引用

ACM International Conference on Multimedia Retrieval (ICMR)

作者： Dong, Pei Wu, Lei Meng, Lei Meng, Xiangxu Shandong Univ Jinan Shandong Peoples R China

ISBN: (纸本)9781450392389

In this paper, we focus on generating photo-realistic images from given text descriptions. Current methods first generate an initial image and then progressively refine it to a high-resolution one. These methods typically indiscriminately refine all granularity features output from the previous stage. However, the ability to express different granularity features in each stage is not consistent, and it is difficult to express precise semantics by further refining the features with poor quality generated in the previous stage. Current methods cannot refine different granularity features independently, resulting in that it is challenging to clearly express all factors of semantics in generated image, and some features even become worse. To address this issue, we propose a Hierarchical Disentangled Representations Generative Adversarial Networks (HDR-GAN) to generate photo-realistic images by explicitly disentangling and individually modeling the factors of semantics in the image. HDR-GAN introduces a novel component called multi-granularity feature disentangled encoder to represent image information comprehensively through explicitly disentangling multi-granularity features including pose, shape and texture. Moreover, we develop a novel Multi-granularity Feature Refinement (MFR) containing a Coarse-grained Feature Refinement (CFR) model and a Fine-grained Feature Refinement (FFR) model. CFR utilizes coarse-grained disentangled representations (e.g., pose and shape) to clarify category information, while FFR employs fine-grained disentangled representations (e.g., texture) to reflect instance-level details. Extensive experiments on two well-studied and publicly available datasets (i.e., CUB-200 and CLEVR-SV) demonstrate the rationality and superiority of our method. [GRAPHICS] .

关键词： text-to-image synthesis Hierarchical Disentangled Representations Generative Adversarial Networks Multi-granularity Features

来源：评论

学校读者我要写书评

暂无评论

Obj-SA-GAN: Object-Driven text-to-image synthesis with Self-Attention Based Full Semantic Information Mining 19th

Obj-SA-GAN: Object-Driven Text-to-Image Synthesis with Self-...

引用

19th Pacific Rim International Conference on Artificial Intelligence (PRICAI)

作者： Li, Ruijun Li, Weihua Yang, Yi Bai, Quan Auckland Univ Technol Auckland 1010 New Zealand Hefei Univ Technol Hefei 230601 Peoples R China Univ Tasmania Hobart Tas 7005 Australia

ISBN: (纸本)9783031208614;9783031208621

In recent years, text-to-image synthesis techniques have made considerable breakthroughs, but the progress is restricted to simple scenes. Such techniques turn out to be ineffective if the text appears complex and contains multiple objects. To address this challenging issue, we propose a novel text-to-image synthesis model called Object-driven Self-Attention Generative Adversarial Network (Obj-SA-GAN), where self-attention mechanisms are utilised to analyse the information with different granularities at different stages, achieving full exploitation of text semantic information from coarse to fine. Complex datasets are used to evaluate the performance of the proposed model. The experimental results explicitly show that our model outperforms the state-of-the-art methods. This is because the proposed Obj-SA-GAN model utilises textual information, which provides a better understanding of complex scenarios.

关键词： text-to-image synthesis Attention Self-Attention Semantic mining GAN

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：