检索结果-内蒙古大学图书馆

Generative adversarial network based on semantic consistency for text-to-image generation

APPLIED INTELLIGENCE 2023年第4期53卷 4703-4716页

作者： Ma, Yue Liu, Li Zhang, Huaxiang Wang, Chunjing Wang, Zekang Shandong Normal Univ Sch Informat Sci & Engn Jinan 250014 Shandong Peoples R China

Although text-to-image generation technology has made significant progress in visually realistic images, the generated images cannot be completely consistent with the texts. In this paper, a novel generative adversarial network based on semantic consistency is proposed to generate semantically consistent and realistic images according to text descriptions. The proposed method explores the semantic consistency between text and image for an efficient cross-modal generation that combines image generation and semantic correlation. A generation network with a hybrid attention is utilized to generate different resolution images, which improves the authenticity of the generated images. In addition, a semantic comparison module is presented to map the texts and the generated images to the same semantic space for comparison through consistency refinement and information classification. Extensive experiments on public benchmark datasets demonstrate that the proposed method outperforms the comparative methods.

关键词： text-to-image generation Generative adversarial network Hybrid attention mechanism Semantic consistency

来源：评论

学校读者我要写书评

暂无评论

Generative adversarial text-to-image generation with style image constraint

引用

MULTIMEDIA SYSTEMS 2023年第6期29卷 3291-3303页

作者： Wang, Zekang Liu, Li Zhang, Huaxiang Liu, Dongmei Song, Yu Shandong Normal Univ Sch Informat Sci & Engn Jinan Peoples R China

Most text-to-image generation works focus on the semantic consistency and neglect the style of the generated image. In this paper, a novel text-to-image generation method is proposed to generate image with style image constraint. In order to provide more comprehensive information by mining long-short-range information dependencies, the multi-group attention module is introduced to capture the multi-scale dependency information in the semantic feature. The adaptive multi-scale attention normalization is adopted to pay the multi-scale style feature attention in the style fusion process. The style information related to semantic feature is filtered out by the style feature attention. This selected style information is transferred to the generated results by aligning the mean and variance of the semantic feature and the style feature. Experiments conducted on common datasets show the validity of the proposed approach.

关键词： text-to-image generation Style fusion Generative adversarial networks Deep feature learning

来源：评论

学校读者我要写书评

暂无评论

LFR-GAN: Local Feature Refinement based Generative Adversarial Network for text-to-image generation

引用

ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS 2023年第6期19卷 1-18页

作者： Deng, Zijun He, Xiangteng Peng, Yuxin Peking Univ Wangxuan Inst Comp Technol Beijing 100871 Peoples R China Peking Univ Natl Key Lab Multimedia Informat Proc Beijing 100871 Peoples R China

text-to-image generation aims to generate images from text descriptions. Its main challenge lies in two aspects: (1) Semantic consistency, i.e., the generated images should be semantically consistent with the input text;and (2) Visual reality, i.e., the generated images should look like real images. To ensure text-image consistency, existing works mainly learn to establish the cross-modal representations via a text encoder and image encoder. However, due to the limited representation capability of the fixed-length embeddings and the flexibility of the free-form text descriptions, the learned text-to-image model is incapable of maintaining the semantic consistency between image local regions and fine-grained descriptions. As a result, the generated images sometimes miss some fine-grained attributes of the generated object, such as the color or shape of a part of the object. To address this issue, this paper proposes a Local Feature Refinement Based Generative Adversarial Network (LFR-GAN), which first divides the text into some independent fine-grained attributes and generates an initial image, then refines the image details based on these attributes. The main contributions are three-fold: (1) An attribute modeling approach is proposed to model the fine-grained text descriptions by mapping them into representations of independent attributes, which provides more fine-grained details for image generation. (2) A local feature refinement approach is proposed to enable the generated image to form a complete reflection of the fine-grained attributes contained in the text description. (3) A multi-stage generation approach is proposed to realize the fine-grained manipulation of complex images progressively, which aims to improve the performance of the refinement and generate photo-realistic images. Extensive experiments on the CUB and Oxford102 datasets show the effectiveness of our LFR-GAN approach in both text-to-image generation and text-guided image manipulation tasks. O

关键词： Local feature refinement text-to-image generation generative adversarial network

来源：评论

学校读者我要写书评

暂无评论

DiverGAN: An Efficient and Effective Single-Stage Framework for Diverse text-to-image generation

引用

NEUROCOMPUTING 2022年 473卷 182-198页

作者： Zhang, Zhenxing Schomaker, Lambert Univ Groningen Fac Sci & Engn Bernoulli Inst NL-9747 Groningen Netherlands

In this paper, we concentrate on the text-to-image synthesis task that aims at automatically producing perceptually realistic pictures from text descriptions. Recently, several single-stage methods have been proposed to deal with the problems of a more complicated multi-stage modular architecture. However, they often suffer from the lack-of-diversity issue, yielding similar outputs given a single textual sequence. To this end, we present an efficient and effective single-stage framework (DiverGAN) to generate diverse, plausible and semantically consistent images according to a natural-language description. DiverGAN adopts two novel word-level attention modules, i.e., a channel-attention module (CAM) and a pixel-attention module (PAM), which model the importance of each word in the given sentence while allowing the network to assign larger weights to the significant channels and pixels semantically aligning with the salient words. After that, Conditional Adaptive Instance-Layer Normalization (CAdaILN) is introduced to enable the linguistic cues from the sentence embedding to flexibly manipulate the amount of change in shape and texture, further improving visual-semantic representation and helping stabilize the training. Also, a dual-residual structure is developed to preserve more original visual features while allowing for deeper networks, resulting in faster convergence speed and more vivid details. Furthermore, we propose to plug a fully-connected layer into the pipeline to address the lack-of-diversity problem, since we observe that a dense layer will remarkably enhance the generative capability of the network, balancing the trade-off between a low-dimensional random latent code contributing to variants and modulation modules that use high-dimensional and textual contexts to strength feature maps. Inserting a linear layer after the second residual block achieves the best variety and quality. Both qualitative and quantitative results on benchmark data sets demonst

关键词： text-to-image generation lack-of-diversity issue single-stage framework generative adversarial network attention mechanism

来源：评论

学校读者我要写书评

暂无评论

Using text-to-image generation for architectural design ideation

引用

INTERNATIONAL JOURNAL OF ARCHITECTURAL COMPUTING 2024年第3期22卷 458-474页

作者： Paananen, Ville Oppenlaender, Jonas Visuri, Aku Univ Oulu Ctr Ubiquitous Comp Pentti Kaiteran Katu 1 Oulu 90014 Finland Elisa Corp Helsinki Finland

text-to-image generation has become very popular in various domains requiring creativity. This article investigates the potential of text-to-image generators in supporting creativity during the early stages of the architectural design process. We conducted a laboratory study with 17 architecture students, who developed a concept for a culture center using three popular text-to-image generators: Midjourney, Stable Diffusion, and DALL-E. Through standardized questionnaires and group interviews, we found that image generation could be a meaningful part of the design process when design constraints are carefully considered. Generative tools support serendipitous discovery of ideas and an imaginative mindset, enriching the design process. We identified several challenges of image generators and provided considerations for software development and educators to support creativity and emphasize designers' imaginative mindset. By understanding the limitations and potential of text-to-image generators, architects and designers can leverage this technology in their design process and education, facilitating innovation and effective communication of concepts.

关键词： Architecture design creativity generative artificial intelligence text-to-image generation

来源：评论

学校读者我要写书评

暂无评论

Social Biases through the text-to-image generation Lens 23

Social Biases through the Text-to-Image Generation Lens

引用

AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES)

作者： Naik, Ranjita Nushi, Besmira Microsoft Redmond WA 98052 USA Microsoft Res Redmond WA USA

ISBN: (纸本)9798400702310

text-to-image (T2I) generation is enabling new applications that support creators, designers, and general end users of productivity software by generating illustrative content with high photorealism starting from a given descriptive text as a prompt. Such models are however trained on massive amounts of web data, which surfaces the peril of potential harmful biases that may leak in the generation process itself. In this paper, we take a multi-dimensional approach to studying and quantifying common social biases as reflected in the generated images, by focusing on how occupations, personality traits, and everyday situations are depicted across representations of (perceived) gender, age, race, and geographical location. Through an extensive set of both automated and human evaluation experiments we present findings for two popular T2I models: DALLE-v2 and Stable Diffusion. Our results reveal that there exist severe occupational biases of neutral prompts majorly excluding groups of people from results for both models. Such biases can get mitigated by increasing the amount of specification in the prompt itself, although the prompting mitigation will not address discrepancies in image quality or other usages of the model or its representations in other scenarios. Further, we observe personality traits being associated with only a limited set of people at the intersection of race, gender, and age. Finally, an analysis of geographical location representations on everyday situations (e.g., park, food, weddings) shows that for most situations, images generated through default location-neutral prompts are closer and more similar to images generated for locations of United States and Germany.

关键词： text-to-image generation representational fairness social biases

来源：评论

学校读者我要写书评

暂无评论

Enhancing Baidu Multimodal Advertisement with Chinese text-to-image generation via Bilingual Alignment and Caption Synthesis 47

Enhancing Baidu Multimodal Advertisement with Chinese Text-t...

引用

47th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

作者： Zhao, Kang Zhao, Xinyu Jin, Zhipeng Yang, Yi Tao, Wen Han, Cong Li, Shuanglong Liu, Lin Baidu Inc Baidu Search Ads Beijing Peoples R China

ISBN: (纸本)9798400704314

Recent advances in generative artificial intelligence have revolutionized information retrieval and content generation, opening up new opportunities for the e-commerce industry. In particular, text-to-image generation models offer a novel approach to guiding the image generation process using natural language input, which is inspiring for multimodal search advertising. Traditional multimodal search ads require advertisers to prepare ad creatives, such as ad images, which is time-consuming and requires uniform image specifications and content quality inspection. To this end, we propose a streamlined generation framework for search ad image creatives. First, we prepare a Chinese image caption model with high-quality image-caption pairs to bootstrap training data refinement. With curated high-quality images and synthesized descriptive captions, we then train a Chinese text-to-image generation model, the largest to date, using SDXL and a 10-billion multimodal text encoder. Specifically, we introduce a two-stage bilingual multimodal representation alignment process to seamlessly integrate the text encoder with the generation model. Extensive experiments validate the effectiveness of our framework, including assessments of image captioning and image generation. The implementation of our framework in Baidu Search Ads shows significant revenue increase, For example, beauty industry ads with generated image creatives achieve a 29% higher click-through rate (CTR).

关键词： text-to-image generation Multimodal Sponsored Search Advertisement image Creatives

来源：评论

学校读者我要写书评

暂无评论

EXTENDING IMPLICIT NEURAL REPRESENTATIONS FOR text-to-image generation 49

EXTENDING IMPLICIT NEURAL REPRESENTATIONS FOR TEXT-TO-IMAGE ...

引用

49th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Liu, Guanming Wei, Zhihua Zhang, Heng Wang, Rui Yuan, Aiquan Liu, Chuanbao Chen, Biao Cao, Guodong Tongji Univ Dept Comp Sci & Technol Shanghai Peoples R China Alibaba Grp Hangzhou Peoples R China

ISBN: (纸本)9798350344868;9798350344851

Implicit neural representations (INRs) have demonstrated their effectiveness in continuous modeling for image signals. However, INRs typically operate in a continuous space, which makes it difficult to integrate the discrete symbols and structures inherent in human language. Despite this, text features carry rich semantic information that is helpful for visual representations, alleviating the demand of INR-based generative models for improvement in diverse datasets. To this end, we propose EIDGAN, an Efficient scale-Invariant Dual-modulated generative adversarial network, extending INRs for text-to-image generation while balancing network's representation power and computation costs. The spectral modulation utilizes Fourier transform to introduce global sentence information into the channel-wise frequency domain of image features. The cross attention modulation, as second-order polynomials incorporating the style codes, introduces local word information while recursively increasing the expressivity of a synthesis network. Benefiting from the column-row entangled bi-line design, EIDGAN enables text-guided generation of any-scale images and semantic extrapolation beyond image boundaries. We conduct experiments on text-to-image tasks based on MS-COCO and CUB datasets, demonstrating competitive performance on INR-based methods.

关键词： generative adversarial network implicit neural representation text-to-image generation style modulation cross attention Fourier transform

来源：评论

学校读者我要写书评

暂无评论

STRUCTURE-AWARE GENERATIVE ADVERSARIAL NETWORK FOR text-to-image generation 30

STRUCTURE-AWARE GENERATIVE ADVERSARIAL NETWORK FOR TEXT-TO-I...

引用

30th IEEE International Conference on image Processing (ICIP)

作者： Chen, Wenjie Ni, Zhangkai Wang, Hanli Tongji Univ Dept Comp Sci & Technol Shanghai Peoples R China Tongji Univ Minist Educ Key Lab Embedded Syst & Serv Comp Shanghai Peoples R China

ISBN: (纸本)9781728198354

text-to-image generation aims at synthesizing photo-realistic images from textual descriptions. Existing methods typically align images with the corresponding texts in a joint semantic space. However, the presence of the modality gap in the joint semantic space leads to misalignment. Meanwhile, the limited receptive field of the convolutional neural network leads to structural distortions of generated images. In this work, a structure-aware generative adversarial network (SaGAN) is proposed for (1) semantically aligning multimodal features in the joint semantic space in a learnable manner;and (2) improving the structure and contour of generated images by the designed content-invariant negative samples. Experimental results show that SaGAN achieves over 30.1% and 8.2% improvements in terms of FID on the datasets of CUB and COCO when compared with the state-of-the-art approaches.

关键词： text-to-image generation generative adversarial network negative data augmentation

来源：评论

学校读者我要写书评

暂无评论

Dynamic Prompt Optimizing for text-to-image generation

Dynamic Prompt Optimizing for Text-to-Image Generation

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Mo, Wenyi Zhang, Tianyu Bai, Yalong Su, Bing Wen, Ji-Rong Yang, Qing Renmin Univ China Gaoling Sch Artificial Intelligence Beijing Peoples R China Beijing Key Lab Big Data Management & Anal Method Beijing Peoples R China Du Xiaoman Technol Beijing Peoples R China

ISBN: (纸本)9798350353006

text-to-image generative models, specifically those based on diffusion models like imagen and Stable Diffusion, have made substantial advancements. Recently, there has been a surge of interest in the delicate refinement of text prompts. Users assign weights or alter the injection time steps of certain words in the text prompts to improve the quality of generated images. However, the success of fine-control prompts depends on the accuracy of the text prompts and the careful selection of weights and time steps, which requires significant manual intervention. To address this, we introduce the Prompt Auto-Editing (PAE) method. Besides refining the original prompts for image generation, we further employ an online reinforcement learning strategy to explore the weights and injection time steps of each word, leading to the dynamic fine-control prompts. The reward function during training encourages the model to consider aesthetic score, semantic consistency, and user preferences. Experimental results demonstrate that our proposed method effectively improves the original prompts, generating visually more appealing images while maintaining semantic alignment. Code is available at this https URL.

关键词： Diffusion Model Prompt Learning Reinforcement Learning text-to-image generation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：