检索结果-内蒙古大学图书馆

How far are we to GPT-4V?Closing the gap to commercial multimodal models with open-source suites

Science China(Information Sciences) 2024年第12期67卷 5-22页

作者： Zhe CHEN Weiyun WANG Hao TIAN Shenglong YE Zhangwei GAO Erfei CUI Wenwen TONG Kongzhi HU Jiapeng LUO Zheng MA Ji MA Jiaqi WANG Xiaoyi DONG Hang YAN Hewei GUO Conghui HE Botian SHI Zhenjiang JIN Chao XU Bin WANG Xingjian WEI Wei LI Wenjian ZHANG Bo ZHANG Pinlong Cai Licheng WEN Xiangchao YAN Min DOU Lewei LU Xizhou ZHU Tong LU Dahua LIN Yu QIAO Jifeng Dai Wenhai WANG State Key Laboratory for Novel Software Technology Nanjing University Shanghai AI Laboratory School of Computer Science Fudan University SenseTime Research Department of Information Engineering The Chinese University of Hong Kong Department of Electronic Engineering Tsinghua University

In this paper, we introduce InternVL 1.5, an open-source multimodal large language model(MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements.(1) Strong vision encoder: we explored a continuous learning strategy for the large-scale vision foundation model — InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs.(2) Dynamic high-resolution: we divide images into tiles ranging from 1 to 40 of 448×448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input.(3) High-quality bilingual dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images,and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in optical character recognition(OCR) and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary commercial models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 multimodal benchmarks. Code and models are available at https://***/OpenGVLab/InternVL.

关键词： multimodal model open-source vision encoder dynamic resolution bilingual dataset

来源：评论

学校读者我要写书评

暂无评论

MMInstruct: a high-quality multi-modal instruction tuning dataset with extensive diversity

引用

Science China(Information Sciences) 2024年第12期67卷 36-51页

作者： Yangzhou LIU Yue CAO Zhangwei GAO Weiyun WANG Zhe CHEN Wenhai WANG Hao TIAN Lewei LU Xizhou ZHU Tong LU Yu QIAO Jifeng Dai School of Computer Science Nanjing University School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University Shanghai AI Laboratory School of Computer Science Fudan University Department of Information Engineering The Chinese University of Hong Kong SenseTime Research Department of Electronic Engineering Tsinghua University

Despite the effectiveness of vision-language supervised fine-tuning in enhancing the performance of vision large language models(VLLMs), existing visual instruction tuning datasets include the following limitations.(1) Instruction annotation quality: despite existing VLLMs exhibiting strong performance,instructions generated by those advanced VLLMs may still suffer from inaccuracies, such as hallucinations.(2) Instructions and image diversity: the limited range of instruction types and the lack of diversity in image data may impact the model's ability to generate diversified and closer to real-world scenarios outputs. To address these challenges, we construct a high-quality, diverse visual instruction tuning dataset MMInstruct,which consists of 973k instructions from 24 domains. There are four instruction types: judgment, multiplechoice, long visual question answering, and short visual question answering. To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3.5, and manual correction. Our instruction generation engine enables semi-automatic, low-cost, and multi-domain instruction generation at 1/6 the cost of manual construction. Through extensive experiment validation and ablation experiments,we demonstrate that MMInstruct could significantly improve the performance of VLLMs, e.g., the model fine-tuning on MMInstruct achieves new state-of-the-art performance on 10 out of 12 benchmarks. The code and data shall be available at https://***/yuecao0119/MMInstruct.

关键词： instruction tuning multi-modal multi-domain dataset vision large language model

来源：评论

学校读者我要写书评

暂无评论

End-to-end differentiability and tensor processing unit computing to accelerate materials’ inverse design

引用

npj Computational Materials 2023年第1期9卷 1072-1083页

作者： Han Liu Yuhan Liu Kevin Li Zhangji Zhao Samuel S.Schoenholz Ekin D.Cubuk Puneet Gupta Mathieu Bauchy SOlids inFormaTics AI-Laboratory(SOFT-AI-Lab) College of Polymer Science and EngineeringSichuan UniversityChengdu 610065China AIMSOLID Research Wuhan 430223China Physics of AmoRphous and Inorganic Solids Laboratory(PARISlab) Department of Civil and Environmental EngineeringUniversity of CaliforniaLos AngelesCA 90095USA Department of Computer Science University of CaliforniaLos AngelesCA 90095USA Google Research Brain TeamMountain ViewCAUSA Department of Electrical and Computer Engineering University of CaliforniaLos AngelesCA 90095USA

Numerical simulations have revolutionized material ***,although simulations excel at mapping an input material to its output property,their direct application to inverse design has traditionally been limited by their high computing cost and lack of ***,taking the example of the inverse design of a porous matrix featuring targeted sorption isotherm,we introduce a computational inverse design framework that addresses these challenges,by programming differentiable simulation on TensorFlow platform that leverages automated end-to-end *** to its differentiability,the simulation is used to directly train a deep generative model,which outputs an optimal porous matrix based on an arbitrary input sorption isotherm ***,this inverse design pipeline leverages the power of tensor processing units(TPU)—an emerging family of dedicated chips,which,although they are specialized in deep learning,are flexible enough for intensive scientific *** approach holds promise to accelerate inverse materials design.

关键词： enough inverse isotherm

来源：评论

学校读者我要写书评

暂无评论

Preference detection of the humanoid robot face based on EEG and eye movement

引用

Neural Computing and Applications 2024年第19期36卷 11603-11621页

作者： Wang, Pengchao Mu, Wei Zhan, Gege Wang, aiping Song, Zuoting Fang, Tao Zhang, Xueze Wang, Junkongshuai Niu, Lan Bin, Jianxiong Zhang, Lihua Jia, Jie Kang, Xiaoyang Laboratory for Neural Interface and Brain Computer Interface Engineering Research Center of AI and Robotics Ministry of Education Shanghai Engineering Research Center of AI and Robotics MOE Frontiers Center for Brain Science State Key Laboratory of Medical Neurobiology Institute of AI and Robotics Academy for Engineering and Technology Fudan University Shanghai200433 China Ji Hua Laboratory Guangdong Province Foshan528200 China Department of Rehabilitation Medicine Huashan Hospital Fudan University Shanghai200040 China Yiwu Research Institute of Fudan University Zhejiang Yiwu322000 China Research Center for Intelligent Sensing Zhejiang Lab Hangzhou311121 China

The face of a humanoid robot can affect the user experience, and the detection of face preference is particularly important. Preference detection belongs to a branch of emotion recognition that has received much attention from researchers. Most of the previous preference detection studies have been conducted based on a single modality. In this paper, we detect face preferences of humanoid robots based on electroencephalogram (EEG) signals and eye movement signals for single modality, canonical correlation analysis fusion modality, and bimodal deep autoencoder (BDAE) fusion modality, respectively. We validated the theory of frontal asymmetry by analyzing the preference patterns of EEG and found that participants had higher alpha wave energy for preference faces. In addition, hidden preferences extracted by EEG signals were better classified than preferences from participants' subjective feedback, and also, the classification performance of eye movement data was improved. Finally, experimental results showed that BDAE multimodal fusion using frontal alpha and beta power spectral densities and eye movement information as features performed best, with the highest average accuracy of 83.13% for the SVM and 71.09% for the KNN. © The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2024.

关键词： Electroencephalography

来源：评论

学校读者我要写书评

暂无评论

Semantic-Aware Pseudo-Labeling for Unsupervised Meta-Learning

引用

IEEE Transactions on Pattern Analysis and Machine Intelligence 2025年第07期47卷 5475-5488页

作者： Ouyang, Tianran Dong, Xingping Ye, Mang Du, Bo Shao, Ling Shen, Jianbing Wuhan University School of Computer Science National Engineering Research Center for Multimedia Software Institute of Artificial Intelligence Hubei Key Laboratory of Multimedia and Network Communication Engineering Wuhan430072 China University of Chinese Academy of Sciences UCAS-Terminus AI Lab Beijing China University of Macau State Key Laboratory of Internet of Things for Smart City Department of Computer and Information Science China

In unsupervised meta-learning, the clustering-based pseudo-labeling approach is an attractive framework, since it is model-agnostic, allowing it to synergize with supervised algorithms to learn from unlabeled data. However, the pseudo-labels suffer from clustering noise and semantic chaos problems, further impacting the effectiveness of meta-learning. In this paper, we analyze and optimize the pseudo-labeling process, including encoding and clustering, aiming to generate semantic- like pseudo-labels to narrow the gap between unsupervised and supervised meta-learning. Firstly, during the encoding, we observe that the embedding space of existing methods lacks clustering-friendly properties, which is the primary reason for clustering noise. To address this issue, we minimize the inter-to-intra-class similarity ratio to generate clustering-friendly embedding features and validate our approach through comprehensive experiments. Then, during the clustering, we find that the semantic quality of pseudo-labels is not adequately controlled, resulting in semantic chaos of pseudo-labels. We propose a semantic-stability index to measure the semantic quality of pseudo-labels quantitatively. Based on this index, we propose the Semantic-aware Pseudo-label Reassignment mechanism to generate semantic- like pseudo-labels for all samples. Our approach is model-agnostic and can easily be integrated into existing supervised methods. To demonstrate its generalization ability, we integrate it into two representative algorithms: MAML and EP. The results on three main few-shot benchmarks clearly show that the proposed method achieves significant improvement compared to state-of-the-art models. Notably, our approach also outperforms the corresponding supervised method in three tasks. The source code will be made publicly available. ©1979-2012 IEEE.

关键词： Latent semantic analysis

来源：评论

学校读者我要写书评

暂无评论

Recent advances in heterogeneous catalysis of solar-driven carbon dioxide conversion

引用

Journal of Environmental Sciences 2024年第6期140卷 165-182页

作者： Jun Xu Farzaneh Arabpour Roghabadi Ying Luo Vahid Ahmadi Qian Wang Zheng Wang Hong He State Key Joint Laboratory of Environment Simulation and Pollution Control Research Center for Eco-Environmental SciencesChinese Academy of SciencesBeijing 100085China College of Chemistry and Chemical Engineering Henan Polytechnic UniversityJiaozuoHenan 454000China Department of Process Engineering Faculty of Chemical EngineeringTarbiat Modares UniversityTehranIran Optoelectronics and Nanophotonics Research Group Faculty of Electrical and Computer EngineeringTarbiat Modares UniversityTehranIran Graduate School of Engineering Nagoya UniversityFuro-choChikusa-kuNagoya 464-8603Japan

Solar-driven carbon dioxide(CO_(2))conversion including photocatalytic(PC),photoelectrochemical(PEC),photovoltaic plus electrochemical(PV/EC)systems,offers a renewable and scalable way to produce fuels and high-value chemicals for environment and energy *** review summarizes the basic fundament and the recent advances in the field of solar-driven CO_(2)*** the visible-light absorption is an important strategy to improve solar energy conversion *** separation and migration of photogenerated charges carriers to surface sites and the surface catalytic processes also determine the photocatalytic *** engineering including co-catalyst loading,defect engineering,morphology control,surface modification,surface phase junction,and Z-scheme photocatalytic system construction,have become fundamental strategies to obtain high-efficiency *** to photocatalysis,these strategies have been applied to improve the conversion efficiency and Faradaic efficiency of typical PEC *** PV/EC systems,the electrode surface structure and morphology,electrolyte effects,and mass transport conditions affect the activity and selectivity of electrochemical CO_(2)***,the challenges and prospects are addressed for the development of solar-driven CO_(2)conversion system with high energy conversion efficiency,high product selectivity and stability.

关键词： Solar-driven CO_(2)reduction Photocatalytic system Photoelectrochemical system Photovoltaic plus electrochemical system Surface/interface engineering

来源：评论

学校读者我要写书评

暂无评论

Large Language Models as Evolutionary Optimizers 13

Large Language Models as Evolutionary Optimizers

引用

13th IEEE Congress on Evolutionary Computation, CEC 2024

作者： Liu, Shengcai Chen, Caishun Qu, Xinghua Tang, Ke Ong, Yew-Soon Southern University of Science and Technology Centre for Frontier Ai Research A∗star Department of Computer Science and Engineering China Centre for Frontier Ai Research A∗star Singapore Tianqiao & Chrissy Chen Institute China Southern University of Science and Technology Guangdong Provincial Key Laboratory of Brain-Inspired Intelligent Computation Department of Computer Science and Engineering China Nanyang Technological University Centre for Frontier Ai Research A∗star College of Computing and Data Science Singapore

ISBN: (纸本)9798350308365

Evolutionary algorithms (EAs) have achieved remarkable success in tackling complex combinatorial optimization problems. However, EAs often demand carefully-designed operators with the aid of domain expertise to achieve satisfactory performance. In this work, we present the first study on large language models (LLMs) as evolutionary combinatorial optimizers. The main advantage is that it requires minimal domain knowledge and human efforts, as well as no additional training of the model. This approach is referred to as LLM-driven EA (LMEA). Specifically, in each generation of the evolutionary search, LMEA instructs the LLM to select parent solutions from current population, and perform crossover and mutation to generate offspring solutions. Then, LMEA evaluates these new solutions and include them into the population for the next generation. LMEA is equipped with a self-adaptation mechanism that controls the temperature of the LLM. This enables it to balance between exploration and exploitation and prevents the search from getting stuck in local optima. We investigate the power of LMEA on the classical traveling salesman problems (TSPs) widely used in combinatorial optimization research. Notably, the results show that LMEA performs competitively to traditional heuristics in finding high-quality solutions on TSP instances with up to 20 nodes. Additionally, we also study the effectiveness of LLM-driven crossover/mutation and the self- adaptation mechanism in evolutionary search. In summary, our results reveal the great potentials of LLMs as evolutionary optimizers for solving combinatorial problems. We hope our research shall inspire future explorations on LLM-driven EAs for complex optimization challenges. © 2024 IEEE.

关键词： Traveling salesman problem

来源：评论

学校读者我要写书评

暂无评论

Improving the Interpretability through Maximizing Mutual Information for EEG Emotion Recognition

引用

IEEE Transactions on Affective Computing 2024年第2期16卷 744-757页

作者： Yang, Hua Philip Chen, C.L. Chen, Bianna Zhang, Tong North Carolina State University Department of Computer Science United States South China University of Technology Guangdong Provincial Key Laboratory of AI Large Model and Intelligent Cognition The School of Computer Science and Engineering Guangzhou510006 China Pazhou Lab Guangzhou510335 China Engineering Research Center The Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human Guangzhou China

Trustworthy Graph Neural Networks (GNNs) for EEG emotion recognition should identify emotions accurately and elucidate corresponding rationales. Current GNNs have achieved notable performance by dynamically modeling emotional connections between EEG channels. However, these GNNs lack interpretability due to the absence of explicit rationale behind their predictions. This paper conducts a comprehensive identification of important EEG channels to enhance the interpretability of EEG emotion recognition from the perspective of mutual information. Specifically, an Adjacency-Explainable Graph Neural Network (AEG) for ante-hoc interpretability is proposed to capture genuine EEG emotional connections, which gives a theoretical guarantee to remove spurious connections. Moreover, a Channel-wise Adaptive Class Activation Mapping Explainer (CACA) for post-hoc interpretability is developed to locate the EEG channels that contribute most to predictions. Experimental results on three datasets, i.e., SEED, SEED-IV, and DREAMER, prove that imbuing training processes with enhanced interpretability ensures significant performance improvements in emotion recognition. Quantitative comparisons of post-hoc interpretability also demonstrate the superiority of CACA. Furthermore, this paper illustrates two potential applications of the proposed methodologies, showing their broader utility and significance. © 2010-2012 IEEE.

关键词： Graph neural networks

来源：评论

学校读者我要写书评

暂无评论

Optical Satellite Networks

Optical Satellite Networks

引用

2023 Optical Fiber Communications Conference and Exhibition, OFC 2023

作者： Chan, Vincent W.S. Massachusetts Institute of Technology Research Laboratory of Electronics Steve Schwarzman College of Computing Ai and Decision System Sector Department of Electrical Engineering and Computer Science United States

We will explore the architecture of optical satellite networks at 100G-1Tbps. The challenge is to architect the system and the network protocols with large bandwidth-delay products and the presence of atmospheric turb... 详细信息

ISBN: (纸本)9781957171180

关键词： Atmospheric turbulence

来源：评论

学校读者我要写书评

暂无评论

Composing Like an Ancient Chinese Poet:Learn to Generate Rhythmic Chinese Poetry

引用

Journal of computer Science & Technology 2023年第6期38卷 1272-1287页

作者：何明陈艳赵洪科刘淇吴乐崔羽曾贵华刘贵全 Department of Electronic Engineering Shanghai Jiao Tong UniversityShanghai 200240China AI Laboratory Lenovo ResearchBeijing 100193China Baidu Incorporated Beijing 100193China College of Management and Economics Tianjin UniversityTianjin 300072China School of Computer Science and Technology University of Science and Technology of ChinaHefei 230027China School of Computer Science and Information Engineering Hefei University of TechnologyHefei 230009China Risk Control Department OPayBeijing 100010China

Automatic generation of Chinese classical poetry is still a challenging problem in artificial ***-cently,Encoder-Decoder models have provided a few viable methods for poetry ***,by reviewing the pri-or methods,two major issues still need to be settled:1)most of them are one-stage generation methods without further polishing;2)they rarely take into consideration the restrictions of poetry,such as tone and ***,some an-cient Chinese poets tended first to write a coarse poem underlying aesthetics and then deliberated its semantics;while oth-ers first create a semantic poem and then refine its *** this basis,in order to better imitate the human creation procedure of poems,we propose a two-stage method(i.e.,restricted polishing generation method)of which each stage fo-cuses on the different aspects of poems(i.e.,semantics and aesthetics),which can produce a higher quality of generated *** this way,the two-stage method develops into two symmetrical generation methods,the aesthetics-to-semantics method and the semantics-to-aesthetics *** particular,we design a sampling method and a gate to formulate the tone and rhyme restrictions,which can further improve the rhythm of the generated *** results demon-strate the superiority of our proposed two-stage method in both automatic evaluation metrics and human evaluation met-rics compared with baselines,especially in yielding consistent improvements in tone and rhyme.

关键词： aesthetics poetry generation polishing semantics two-stage method

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：