检索结果-内蒙古大学图书馆

text-to-image synthesis With Generative Models: Methods, Datasets, Performance Metrics, Challenges, and Future Direction

引用

IEEE ACCESS 2024年 12卷 24412-24427页

作者： Alhabeeb, Sarah K. Al-Shargabi, Amal A. Qassim Univ Coll Comp Dept Informat Technol Buraydah 51452 Saudi Arabia

text-to-image synthesis, the process of turning words into images, opens up a world of creative possibilities, and meets the growing need for engaging visual experiences in a world that is becoming more image-based. As machine learning capabilities expanded, the area progressed from simple tools and systems to robust deep learning models that can automatically generate realistic images from textual inputs. Modern, large-scale text-to-image generation models have made significant progress in this direction, producing diversified and high-quality images from text description prompts. Although several methods exist, Generative Adversarial Networks (GANs) have long held a position of prominence. However, diffusion models have recently emerged, with results much beyond those achieved by GANs. This study offers a concise overview of text-to-image generative models by examining the existing body of literature and providing a deeper understanding of this topic. This will be accomplished by providing a concise summary of the development of text-to-image synthesis, previous tools and systems employed in this field, key types of generative models, as well as an exploration of the relevant research conducted on GANs and diffusion models. Additionally, the study provides an overview of common datasets utilized for training the text-to-image model, compares the evaluation metrics used for evaluating the models, and addresses the challenges encountered in the field. Finally, concluding remarks are provided to summarize the findings and implications of the study and open issues for further research.

关键词： Deep learning diffusion model generative models generative adversarial network text-to-image synthesis

来源：评论

学校读者我要写书评

暂无评论

text-to-image synthesis: Starting composite from the foreground content

引用

INFORMATION SCIENCES 2022年 607卷 1265-1285页

作者： Zhang, Zhiqiang Zhou, Jinjia Yu, Wenxin Jiang, Ning Hosei Univ Grad Sch Sci & Engn Tokyo Japan Southwest Univ Sci & Technol Sch Comp Sci & Technol Mianyang Sichuan Peoples R China

Recently, text-to-image synthesis has become a hot issue in computer vision and has been widely concerned. Many methods have achieved encouraging results in this field at present, but it is still a great challenge to improve the quality of the synthesized image further. In this paper, we propose a multi-stage synthesis method, which starts composite from the foreground content. The whole synthesis process is divided into three stages. The first stage generates the foreground results, and the third stage synthesizes the final image results. The second stage results include two situations: one is to continue to synthesize the foreground results;the other is to synthesize the image results with background information. Experiments demonstrate that the method of continuing to generate the foreground results in the second stage can achieve better results on the Caltech-UCSD Birds (CUB) and Oxford-102 datasets, while the way of synthesizing foreground results only in the first stage can obtain better performance on the Microsoft Common Objects in Context (MS COCO) dataset. Besides, our synthesized results on the three datasets are subjectively more realistic with better detail processing. It also outperforms most existing methods in quantitative comparison results, which demonstrates the effectiveness and superiority of our method. (C) 2022 Elsevier Inc. All rights reserved.

关键词： text-to-image synthesis Generative adversarial networks Computer vision Deep learning

来源：评论

学校读者我要写书评

暂无评论

GACnet-text-to-image synthesis With Generative Models Using Attention Mechanisms With Contrastive Learning

引用

IEEE ACCESS 2024年 12卷 9572-9585页

作者： Habib, Md. Ahsan Wadud, Md. Anwar Hussen Pinky, Lubna Yeasmin Talukder, Mehedi Hasan Rahman, Mohammad Motiur Mridha, M. F. Okuyama, Yuichi Shin, Jungpil Mawlana Bhashani Sci & Technol Univ Dept Comp Sci & Engn Tangail 1902 Bangladesh Bangladesh Univ Business & Technol Dept Comp Sci & Engn Dhaka 1216 Bangladesh Amer Int Univ Bangladesh Dept Comp Sci Dhaka 1229 Bangladesh Univ Aizu Sch Comp Sci & Engn Aizu Wakamatsu 9650006 Japan

The generation of high-quality images from textual descriptions is a challenging task in computer vision and natural language processing. The goal of text-to-image synthesis, a current topic of research, is to produce excellent images from written descriptions. This study proposes a hybrid approach to evaluating a dataset consisting of various text-image pairs by efficiently combining conditional generative adversarial networks (C-GAN), attention mechanisms, and contrastive learning (C-GAN+ATT+CL). We suggest a two-step method to improve image quality that starts by utilizing generative adversarial networks (GANs) with attention mechanisms to create low-resolution images and then contrastive learning to improve. Contrastive learning modules train on a separate dataset of high-resolution pictures;GANs learn on datasets of low-resolution text and image pairs. The Conditional GAN with Attention Mechanism and Contrastive Learning Method provides state-of-the-art performance in terms of image quality, diversity, and visual realism, among the several methods. The results of this study demonstrate that the proposed approach works better than all other methods, achieving an Inception Score (IS) of 35.23, a Frechet Inception Distance (FID) of 18.2, and an R-Precision of 89.14. Our findings demonstrate that our "C-GAN+ATT+CL" approach significantly improves image quality and diversity and offers exciting paths for further study.

关键词： Generative adversarial networks Generators image synthesis Training Tokenization Computer science text processing image processing text-to-image synthesis generative adversarial networks C-GAN attention mechanism contrastive learning technique consistency

来源：评论

学校读者我要写书评

暂无评论

Adaptive multi-text union for stable text-to-image synthesis learning

引用

PATTERN RECOGNITION 2024年 152卷

作者： Zhou, Yan Qian, Jiechang Zhang, Huaidong Xu, Xuemiao Sun, Huajie Zeng, Fanzhi Zhou, Yuexia South China Univ Technol Guangzhou 510000 Peoples R China Foshan Univ Foshan 528000 Peoples R China

Generative Adversarial Networks (GANs) have significantly boosted the performance of text -to -image generation tasks in recent years. To train the generator, losses like reconstruction loss or adversarial loss between the generated image and ground truth image are widely adopted by recent works. These losses are all built over one assumption: the given text descriptions can describe the corresponding image perfectly. Unfortunately, this assumption is not satisfied in many cases, especially in datasets like COCO with complicated scenes due to the variance of annotator experience and focal point. This paper addresses this issue by proposing a multi -text -toimage training framework, which adaptively adjusts the weights of all the text descriptions corresponding to a specific image to generate the union description features. With the union description features, the generator can generate more visual-consistent images and mitigate the negative optimization caused by incomplete or inconsistent text descriptions. To better measure the similarity between generated images and multi -text descriptions, we also reformulate the process of multi -modal matching loss to better measure the similarity between image and multi -text descriptions. Extensive experiments on relevant benchmarks CUB and COCO prove the proposed method's effectiveness and superiority compared to state -of -the -art methods.

关键词： Adaptive multi-text union learning text-to-image synthesis Cross-modal generation

来源：评论

学校读者我要写书评

暂无评论

ISF-GAN: Imagine, Select, and Fuse with GPT-based text Enrichment for text-to-image synthesis

引用

ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS 2024年第7期20卷 1-17页

作者： Sheng, Yefei Tao, Ming Wang, Jie Bao, Bing-Kun Nanjing Univ Posts & Telecommun Coll Telecommun & Informat Engn Nanjing Peoples R China Nanjing Univ Posts & Telecommun Sch Comp Sci Nanjing Peoples R China

text-to-image synthesis aims to generate an accurate and semantically consistent image from a given text description. However, it is difficult for existing generative methods to generate semantically complete images from a single piece of text. Some works try to expand the input text to multiple captions via retrieving similar descriptions of the input text from the training set but still fail to fill in missing image semantics. In this article, we propose a GAN-based approach to Imagine, Select, and Fuse for text-to-image synthesis, named ISF-GAN. The proposed ISF-GAN contains Imagine Stage and Select and Fuse Stage to solve the above problems. First, the Imagine Stage proposes a text completion and enrichment module. This module guides a GPT-based model to enrich the text expression beyond the original dataset. Second, the Select and Fuse Stage selects qualified text descriptions and then introduces a cross-modal attentional mechanism to interact these different sentence embeddings with the image features at different scales. In short, our proposed model enriches the input text information for completing missing semantics and introduces a cross-modal attentional mechanism to maximize the utilization of enriched text information to generate semantically consistent images. Experimental results on CUB, Oxford-102, and CelebA-HQ datasets prove the effectiveness and superiority of the proposed network. Code is available at https://***/Feilingg/ISF-GAN

关键词： text-to-image synthesis adversarial learning generative pre-trained model

来源：评论

学校读者我要写书评

暂无评论

Semantic Distance Adversarial Learning for text-to-image synthesis

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2024年 26卷 1255-1266页

作者： Yuan, Bowen Sheng, Yefei Bao, Bing-Kun Chen, Yi-Ping Phoebe Xu, Changsheng Nanjing Univ Posts & Telecommun Coll Telecommun & Informat Engn Nanjing 210003 Peoples R China Peng Cheng Lab Shenzhen 518000 Peoples R China La Trobe Univ Melbourne Vic 3086 Australia Univ Chinese Acad Sci Beijing 101408 Peoples R China Chinese Acad Sci CASIA Inst Automat State Key Lab Multimodal Artificial Intelligence S Beijing 101408 Peoples R China

text-to-image (T2I) synthesis is a cross-modality task that requires a text description as input to generate a realistic and semantically consistent image. To guarantee semantic consistency, previous studies regenerate text descriptions from synthetic images and align them with the given descriptions. However, the existing redescription modules lack explicit modeling of their training objectives, which is crucial for reliable measurement of semantic distance between redescriptions and given text inputs. Consequently, the aligned text redescriptions suffer from training bias caused by the emergence of adversarial image samples, unseen semantics, and mistaken contents from low-quality synthesized images. To this end, we propose a SEMantic distance Adversarial learning (SEMA) framework for text-to-image synthesis which strengthens semantic consistency from two aspects: 1) We introduce adversarial learning between the image generator and the text redescription module to mutually promote or demote the quality of generated image or text instances. This learning model ensures accurate redescription of image contents, thus diminishing the generation of adversarial image samples. 2) We introduce two-fold semantic distance discrimination (SEM distance) to characterize semantic relevance between matching text or image pairs. The unseen semantics and mistaken contents will be penalized with a large SEM distance. The proposed discrimination method also simplifies the model training process with no need to optimize multiple discriminators. Experimental results on CUB Birds 200 and MS-COCO datasets show that the proposed model outperforms the state-of-the-art methods.

关键词： text-to-image synthesis adversarial learning cycle consistency

来源：评论

学校读者我要写书评

暂无评论

Word self-update contrastive adversarial networks for text-to-image synthesis

引用

NEURAL NETWORKS 2023年第1期167卷 433-444页

作者： Xiao, Jian Sun, Yiwen Bi, Xiaojun Harbin Engn Univ Coll Informat & Commun Engn Harbin 150001 Peoples R China Peking Univ Inst Artificial Intelligence Beijing 100871 Peoples R China Minzu Univ China Sch Informat Engn Beijing 100081 Peoples R China Minzu Univ China Key Lab Ethn Language Intelligent Anal & Secur Gov MOE Beijing 100081 Peoples R China

Synthesizing realistic fine-grained images from text descriptions is a significant computer vision task. Although many GANs-based methods have been proposed to solve this task, generating high -quality images consistent with text information remains a difficult problem. These existing GANs-based methods ignore important words due to the use of fixed initial word features in generator, and neglect to learn semantic consistency between images and texts for discriminators. In this article, we propose a novel attentional generation and contrastive adversarial framework for fine-grained text-to-image synthesis, termed as Word Self-Update Contrastive Adversarial Networks (WSC-GAN). Specifically, we introduce a dual attention module for modeling color details and semantic information. With a new designed word self-update module, the generator can leverage visually important words to compute attention maps in the feature synthesis module. Furthermore, we contrive multi-branch contrastive discriminators to maintain better consistency between the generated image and text description. Two novel contrastive losses are proposed for our discriminators to impose image-sentence and image-word consistency constraints. Extensive experiments on CUB and MS-COCO datasets demonstrate that our method achieves better performance compared with state-of-the-art methods.(c) 2023 Elsevier Ltd. All rights reserved.

关键词： text-to-image synthesis Contrastive learning Word self-attention Dual attention generator

来源：评论

学校读者我要写书评

暂无评论

Enhanced text-to-image synthesis With Self-Supervision

引用

IEEE ACCESS 2023年 11卷 39508-39519页

作者： Tan, Yong Xuan Lee, Chin Poo Neo, Mai Lim, Kian Ming Lim, Jit Yan Multimedia Univ Fac Informat Sci & Technol Jalan Ayer Keroh Lama Melaka 75450 Malaysia Multimedia Univ Fac Creat Multimedia Cyberjaya Selangor Malaysia

The task of text-to-image synthesis is a difficult challenge, especially when dealing with low-data regimes, where the number of training samples is limited. In order to address this challenge, the Self-Supervision text-to-image Generative Adversarial Networks (SS-TiGAN) has been proposed. The method employs a bi-level architecture, which allows for the use of self-supervision to increase the number of training samples by generating rotation variants. This, in turn, maximizes the diversity of the model representation and enables the exploration of high-level object information for more detailed image construction. In addition to the use of self-supervision, SS-TiGAN also investigates various techniques to address the stability issues that arise in Generative Adversarial Networks. By implementing these techniques, the proposed SS-TiGAN has achieved a new state-of-the-art performance on two benchmark datasets, Oxford-102 and CUB. These results demonstrate the effectiveness of the SS-TiGAN method in synthesizing high-quality, realistic images from text descriptions under low-data regimes.

关键词： Semantics Generators image synthesis Generative adversarial networks Computer architecture Visualization text mining text-to-image synthesis generative model GAN self-supervised learning generative adversarial networks

来源：评论

学校读者我要写书评

暂无评论

Multi-scale dual-modal generative adversarial networks for text-to-image synthesis

引用

MULTIMEDIA TOOLS AND APPLICATIONS 2023年第10期82卷 15061-15077页

作者： Jiang, Bin Huang, Yun Huang, Wei Yang, Chao Xu, Fangqiang Hunan Univ Coll Comp Sci & Elect Engn Changsha 410082 Hunan Peoples R China

Generating images from text descriptions is a challenging task due to the natural gap between the textual and visual modalities. Despite the promising results of existing methods, they suffer from two limitations: (1) focus more on the image semantic information while fails to fully explore the texture information;(2) only consider to model the correlation between words and image with a fixed scale, thus decreases the diversity and discriminability of the network representations. To address above issues, we propose a Multi-scale Dual-modal Generative Networks (MD-GAN). The core components of MD-GAN are the dual-modal modulation attention (DMA) and the multi-scale consistency discriminator (MCD). The DMA includes two blocks: the textual guiding module that captures the correlation between images and text descriptions to rectify the image semantic content, and the channel sampling module that adjusts image texture by selectively aggregating the channel-wise information on spatial space. In addition, the MCD constructs the correlation between text and image region of various sizes, enhancing the semantic consistency between text and images. Extensive experiments on CUB and MS-COCO datasets show the superiority of MD-GAN over state-of-the-art methods.

关键词： text-to-image synthesis Generative adversarial network Cross-modal Attention mechanism

来源：评论

学校读者我要写书评

暂无评论

Recent Advances in text-to-image synthesis: Approaches, Datasets and Future Research Prospects

引用

IEEE ACCESS 2023年 11卷 88099-88115页

作者： Tan, Yong Xuan Lee, Chin Poo Neo, Mai Lim, Kian Ming Lim, Jit Yan Alqahtani, Ali Multimedia Univ Fac Informat Sci & Technol Melaka Malaysia Multimedia Univ Fac Creat Multimedia Cyberjaya Selangor Malaysia King Khalid Univ Dept Comp Sci Abha Saudi Arabia King Khalid Univ Ctr Artificial Intelligence CAI Abha 61421 Saudi Arabia

text-to-image synthesis is a fascinating area of research that aims to generate images based on textual descriptions. The main goal of this field is to generate images that match the given textual description in terms of both semantic consistency and image realism. While text-to-image synthesis has shown remarkable progress in recent years, it still faces several challenges, mainly related to the level of image realism and semantic consistency. To address these challenges, various approaches have been proposed, which mainly rely on Generative Adversarial Networks (GANs) for optimal performance. This paper provides a review of the existing text-to-image synthesis approaches, which are categorized into four groups: image realism, multiple scene, semantic enhancement, and style transfer. In addition to discussing the existing approaches, this paper also reviews the widely used datasets for text-to-image synthesis, including Oxford-102, CUB-200-2011, and COCO. The evaluation metrics used in this field are also discussed, including Inception Score, Frechet Inception Distance, Structural Similarity Index, R-precision, Visual-Semantic Similarity, and Semantic Object Accuracy. The paper also offers a compilation of the performance of existing works in the field.

关键词： text-to-image synthesis generative model GAN generative adversarial networks review survey

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：