检索结果-内蒙古大学图书馆

text to image generation with Bidirectional Multiway Transformers

COMPUTATIONAL VISUAL MEDIA 2025年第2期11卷 405-422页

作者： Bao, Hangbo Dong, Li Piao, Songhao Wei, Furu Harbin Inst Technol Dept Comp Sci & Technol Harbin 150001 Peoples R China Microsoft Res Asia Beijing 100080 Peoples R China

In this study, we explore the potential of Multiway Transformers for text-to-image generation to achieve performance improvements through a concise and efficient decoupled model design and the inference efficiency provided by bidirectional encoding. We propose a method for improving the image tokenizer using pretrained Vision Transformers. Next, we employ bidirectional Multiway Transformers to restore the masked visual tokens combined with the unmasked text tokens. On the MS-COCO benchmark, our Multiway Transformers outperform vanilla Transformers, achieving superior FID scores and confirming the efficacy of the modality-specific parameter computation design. Ablation studies reveal that the fusion of visual and text tokens in bidirectional encoding contributes to improved model performance. Additionally, our proposed tokenizer outperforms VQGAN in image reconstruction quality and enhances the text-to-image generation results. By incorporating the additional CC-3M dataset for intermediate finetuning on our model with 688M parameters, we achieve competitive results with a finetuned FID score of 4.98 on MS-COCO.

关键词： Visualization Transformers text to image Training Encoding Decoding Computer vision Vocabulary Autoencoders Iterative methods text to image generation VQ-VAE Transformer generative models

来源：评论

学校读者我要写书评

暂无评论

AI Innovator: text to image generation using GAN

AI Innovator: Text to Image Generation using GAN

引用

2024 IEEE International Students' Conference on Electrical, Electronics and Computer Science, SCEECS 2024

作者： Nimbarte, Mrudula Hussain, Alfiya Budhlani, Kanak Bansod, Chirag Ramdham, Mayur Kanhe, Om Khullar, Amit S.B. Jain Institute of Technology Management and Research Department of Computer Science and Engineering MS Nagpur India

ISBN: (纸本)9798350348460

This review provides a comprehensive review of the latest approaches and advances in text-to-image processing using artificial neural networks (GANs). The work under review uses GAN architecture, where generators and classifiers work together to extract corresponding images from descriptions. This research includes a comprehensive review of the GAN framework, delving into the complexities of the generator and discrimination during training. The project solves problems related to converting text input into visual images using a combination of natural language processing and computer vision. This article reviews the main concepts of the architecture as well as the use of advanced neural networks for text understanding and image synthesis. Particular attention is paid to the optimization process the generator uses to improve the quality and integrity of the resulting images. Additionally, the review highlights the evolution of different labels and discusses their implications for real-world applications. © 2024 IEEE.

关键词： Adversarial Training Computer Vision Generative Adversarial Networks (GANs) image Synthesis Multimodal Learning Natural Language Processing (NLP) Neural Network Architecture text to image generation

来源：评论

学校读者我要写书评

暂无评论

Offline Evaluation of Set-Based text-to-image generation 2024

Offline Evaluation of Set-Based Text-to-Image Generation

引用

2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region

作者： Arabzadeh, Negar Diaz, Fernando He, Junfeng Univ Waterloo Waterloo ON Canada Google Pittsburgh PA USA Google Inc Mountain View CA USA Google Montreal PQ Canada

ISBN: (纸本)9798400707247

text-to-image (TTI) systems often support people during ideation, the early stages of a creative process when exposure to a broad set of relevant images can help explore the design space. Since ideation is an important subclass of TTI tasks, understanding how to quantitatively evaluate TTI systems according to how well they support ideation is crucial to promoting research and development for these users. However, existing evaluation metrics for TTI remain focused on distributional similarity metrics like Frechet Inception Distance (FID). We take an alternative approach and, based on established methods from ranking evaluation, develop TTI evaluation metrics with explicit models of how users browse and interact with sets of spatially arranged generated images. Our proposed offline evaluation metrics for TTI not only capture how relevant generated images are with respect to the user's ideation need but also take into consideration the diversity and arrangement of the set of generated images. We analyze our proposed family of TTI metrics using human studies on image grids generated by three different TTI systems based on subsets of the widely used benchmarks such as MS-COCO captions and Localized Narratives as well as prompts used in naturalistic settings. Our results demonstrate that grounding metrics in how people use systems is an important and understudied area of benchmark design.

关键词： text to image generation Offline Evaluation User modelling

来源：评论

学校读者我要写书评

暂无评论

Visual question answering based evaluation metrics for text-to-image generation

Visual question answering based evaluation metrics for text-...

引用

IEEE International Symposium on Circuits and Systems (ISCAS)

作者： Miyamoto, Mizuki Morita, Ryugo Zhou, Jinjia Hosei Univ Tokyo Japan

ISBN: (纸本)9798350330991;9798350331004

text-to-image generation and text-guided image manipulation have received considerable attention in the field of image generation tasks. However, the mainstream evaluation methods for these tasks have difficulty in evaluating whether all the information from the input text is accurately reflected in the generated images, and they mainly focus on evaluating the overall alignment between the input text and the generated images. This paper proposes new evaluation metrics that assess the alignment between input text and generated images for every individual object. Firstly, according to the input text, chatGPT is utilized to produce questions for the generated images. After that, we use Visual Question Answering(VQA) to measure the relevance of the generated images to the input text, which allows for a more detailed evaluation of the alignment compared to existing methods. In addition, we use Non-Reference image Quality Assessment(NR-IQA) to evaluate not only the text-image alignment but also the quality of the generated images. Experimental results show that our proposed evaluation approach is the superior metric that can simultaneously assess finer text-image alignment and image quality while allowing for the adjustment of these ratios.

关键词： neural networks machine learning image generation text-guided image manipulation text to image generation evaluation metrics

来源：评论

学校读者我要写书评

暂无评论

Utilizing Generative AI for text-to-image generation 15

Utilizing Generative AI for Text-to-Image Generation

引用

15th International Conference on Computing Communication and Networking Technologies, ICCCNT 2024

作者： Vinothkumar, S. Varadhaganapathy, S. Shanthakumari, R. Dhanushya, S. Guhan, S. Krisvanth, P. Kongu Engineering College Department of Information Technology Erode India

ISBN: (纸本)9798350370249

With the rapid evolution of generative artificial intelligence (AI), this research project delves into the realm of text to image generation. Leveraging advanced neural network architectures, the study explores the synthesis of visual content from textual descriptions. The project not only investigates the technical intricacies of the generative models involved but also delves into the potential applications across various domains such as creative content creation, design, and multimedia enhancement. The methodology encompasses a comprehensive examination of state-of-the-art techniques in text to image generation, analyzing their strengths, weaknesses, and advancements. Natural language processing (NLP) plays a pivotal role in capturing the semantic nuances embedded in textual inputs, enabling the models to create visually compelling and contextually accurate images. The comparative analysis considers factors such as model performance metrics, scalability, and adaptability to diverse textual inputs. As the project unfolds, it addresses ethical considerations inherent in image synthesis, emphasizing the importance of responsible AI practices and the need for validation mechanisms to ensure the generated images align with the intended context. The paper concludes with a discussion on potential applications of text to image generation in fields like content creation, virtual environments, and educational materials, highlighting the transformative impact of generative AI in shaping the future of visual content creation. © 2024 IEEE.

关键词： creative content creation design Generative AI image synthesis multimedia enhancement natural language processing neural network architectures responsible AI text to image generation validation mechanisms

来源：评论

学校读者我要写书评

暂无评论

Perceptions of generative AI in the architectural profession in Egypt: opportunities, threats, concerns for the future, and steps to improve

引用

AI & SOCIETY 2025年 1-29页

作者： Elrawy, Sara Wagdy, Bahaa Helwan Univ Fac Fine Arts Architecture Dept Cairo Egypt

Generative AI has seen significant advances, particularly in text-to-image, with the potential to revolutionize industries, especially in creative fields such as art and design. This innovation is especially important in architecture, where idea visualization is critical. text-to-image tools, a form of generative AI, enable architects and designers to visually bring their concepts to life. The study explores the impact of prompt-based AI generation on architecture, asking whether it is enhancing efficiency, creativity, and sustainability or threatening to replace architects. To address concerns about the role of AI in the profession, the research examines the perceptions of architecture professionals in Egypt. The authors conducted a survey and interviews with industry experts to assess the transformative impacts of AI on architecture. The findings reveal a strong awareness of AI's potential to enhance design quality and project outcomes, although some concerns about job prospects and control over AI outputs persist. Small firms view AI as vital for optimizing operations and attracting clients. Overall, AI shows promise in conceptualization and visualization, enhancing creativity and efficiency, with architects needing to adapt to AI as a tool for innovation rather than a competitor. Finally, the study proposes a roadmap for improving the use of AI in architecture.

关键词： AI text to image generation Prompt Automation Job loss

来源：评论

学校读者我要写书评

暂无评论

AnyScene: Customized image Synthesis with Composited Foreground

AnyScene: Customized Image Synthesis with Composited Foregro...

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Chen, Ruidong Wang, Lanjun Nie, Weizhi Zhang, Yongdong Liu, An-An Tianjin Univ Tianjin Peoples R China Univ Sci & Technol China Hefei Peoples R China

ISBN: (纸本)9798350353013;9798350353006

Recent advancements in text-to-image technology have significantly advanced the field of image customization. Among various applications, the task of customizing diverse scenes for user-specified composited elements holds great application value but has not been extensively explored. Addressing this gap, we propose AnyScene, a specialized framework designed to create varied scenes from composited foreground using textual prompts. AnyScene addresses the primary challenges inherent in existing methods, particularly scene disharmony due to a lack of foreground semantic understanding and distortion of foreground elements. Specifically, we develop a foreground injection module that guides a pre-trained diffusion model to generate cohesive scenes in visual harmony with the provided foreground. To enhance robust generation, we implement a layout control strategy that prevents distortions of foreground elements. Furthermore, an efficient image blending mechanism seamlessly reintegrates foreground details into the generated scenes, producing outputs with overall visual harmony and precise foreground details. In addition, we propose a new benchmark and a series of quantitative metrics to evaluate this proposed image customization task. Extensive experimental results demonstrate the effectiveness of AnyScene, which confirms its potential in various applications.

关键词： generative model personalized image generation text to image generation

来源：评论

学校读者我要写书评

暂无评论

text-to-image Synthesis using stackGAN

Text-to-Image Synthesis using stackGAN

引用

2023 IEEE Global Conference on Information Technologies and Communications, GCITC 2023

作者： Sahithi, Y. Lakshmi Sunny, N. Deepak, M.M.L. Amrutha, S. Vr Siddhartha Engineering College Andhra Pradesh India

ISBN: (纸本)9798350308167

In text-to-image conversion, it can be challenging to produce visuals of a high caliber from written descriptions. Examples created using traditional text to image approaches can usually convey the meaning of the descriptions given, but they are deficient in important details and vibrant object aspects. The creation of photo-realistic graphics from text has several applications, including photo editing, computer-aided design, gaming, virtual reality, and medical imaging. This project suggests using StackGANs, or stacked generative adversarial networks, to create accurate images depending on the text given. There are two phases to StackGAN. Stage-I GAN generates low-resolution representations of the object by sketching its basic form and colors from the text provided. The Stage-II GAN uses text descriptions and Stage-I discoveries as inputs to create high-resolution images with lifelike qualities. © 2023 IEEE.

关键词： Deep learning High resolution image StackGAN text to image generation virtual reality

来源：评论

学校读者我要写书评

暂无评论

image Attribute Modification Based on text Guidance 1

引用

16th China Conference on Wireless Sensor Networks (CWSN)

作者： Zhao, Liang Li, Xiaoyuan Fu, Chunjiang Chen, Zhikui Dalian Univ Technol Dalian Peoples R China

ISBN: (数字)9789811983504

ISBN: (纸本)9789811983498;9789811983504

The goal of this paper is to manipulate image attributes using text description. Although many methods can synthesize images with new properties from text, they cannot fully preserve the text independent content of the original image. There are two major limitations: (1) Some important details in image sub-region will be lost in the process of image modification;(2) Compared with the original image, the shape and edge of the object in the modified image will be more blurred. Therefore, we propose a novel framework edge aware generative adversarial network (EA-GAN) that uses edge information to guide image modification, which ensures the network's ability to identify local regions and realizes accurate modification of image sub-regions. At the same time, an edge reconstruction loss (ERLoss) is added to the generator to constrain the generation of edges, generate sharper edges, and improve the clarity of the image. The experimental data on the CUB and Oxford-102 datasets show that the algorithm used in this paper can well distinguish the corresponding image features in the conditional text, and modify the image attribute of specific regions in the image.

关键词： text to image generation Generative adversarial network image modification

来源：评论

学校读者我要写书评

暂无评论

PerceptionGAN: Real-world image Construction from Provided text through Perceptual Understanding 9

PerceptionGAN: Real-world Image Construction from Provided T...

引用

Joint 9th International Conference on Informatics, Electronics and Vision (ICIEV) / 4th International Conference on Imaging, Vision and Pattern Recognition (ICIVPR)

作者： Garg, Kanish Singh, Ajeet Kumar Herremans, Dorien Lall, Brejesh Indian Inst Technol Delhi Delhi India Singapore Univ Technol & Design Singapore Singapore

ISBN: (纸本)9781728193311

Generating an image from a provided descriptive text is quite a challenging task because of the difficulty in incorporating perceptual information (object shapes, colors, and their interactions) along with providing high relevancy related to the provided text. Current methods first generate an initial low-resolution image, which typically has irregular object shapes, colors, and interaction between objects. This initial image is then improved by conditioning on the text. However, these methods mainly address the problem of using text representation efficiently in the refinement of the initially generated image, while the success of this refinement process depends heavily on the quality of the initially generated image, as pointed out in the Dynamic Memory Generative Adversarial Network (DM-GAN) paper. Hence, we propose a method to provide good initialized images by incorporating perceptual understanding in the discriminator module. We improve the perceptual information at the first stage itself, which results in significant improvement in the final generated image. In this paper, we have applied our approach to the novel StackGAN architecture. We then show that the perceptual information included in the initial image is improved while modeling image distribution at multiple stages. Finally, we generated realistic multi-colored images conditioned by text. These images have good quality along with containing improved basic perceptual information. More importantly, the proposed method can be integrated into the pipeline of other state-of-the-art text-based-image-generation models such as DM-GAN and AttnGAN to generate initial low-resolution images. We also worked on improving the refinement process in StackGAN by augmenting the third stage of the generator-discriminator pair in the StackGAN architecture. Our experimental analysis and comparison with the state-of-the-art on a large but sparse dataset MS COCO further validate the usefulness of our proposed approach. Cont

关键词： Deep Learning GAN MS COCO text to image generation PerceptionGAN Captioner Loss

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：