检索结果-内蒙古大学图书馆

arXiv 2024年

作者： Liu, Guoshan Yin, Hailong Zhu, Bin Chen, Jingjing Ngo, Chong-Wah Jiang, Yu-Gang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University China Shanghai Collaborative Innovation Center on Intelligent Visual Computing China Singapore Management University Singapore

The growing interest in generating recipes from food images has drawn substantial research attention in recent years. Existing works for recipe generation primarily utilize a two-stage training method—first predicting ingredients from a food image and then generating instructions from both the image and ingredients. Large Multi-modal Models (LMMs), which have achieved notable success across a variety of vision and language tasks, shed light on generating both ingredients and instructions directly from images. Nevertheless, LMMs still face the common issue of hallucinations during recipe generation, leading to suboptimal performance. To tackle this issue, we propose a retrieval augmented large multimodal model for recipe generation. We first introduce Stochastic Diversified Retrieval Augmentation (SDRA) to retrieve recipes semantically related to the image from an existing datastore as a supplement, integrating them into the prompt to add diverse and rich context to the input image. Additionally, Self-Consistency Ensemble Voting mechanism is proposed to determine the most confident prediction recipes as the final output. It calculates the consistency among generated recipe candidates, which use different retrieval recipes as context for generation. Extensive experiments validate the effectiveness of our proposed method, which demonstrates state-of-the-art (SOTA) performance in recipe generation on the Recipe1M dataset. © 2024, CC BY-NC-ND.

关键词： Food ingredients

来源：评论

学校读者我要写书评

暂无评论

MFAE: Masked frame-level autoencoder with hybrid-supervision for low-resource music transcription

MFAE: Masked frame-level autoencoder with hybrid-supervision...

引用

IEEE International Conference on Multimedia and Expo (ICME)

作者： Yulun Wu Jiahao Zhao Yi Yu Wei Li School of Computer Science and Technology Fudan University Shanghai China Shanghai Key Laboratory of Intelligent Information Processing Fudan University Shanghai China

Automantic Music Transcription (AMT) is an essential topic in music information retrieval (MIR), and it aims to transcribe audio recordings into symbolic representations. Recently, large-scale piano datasets with high-quality notations have been proposed for high-resolution piano transcription, which resulted in domain-specific AMT models achieved state-of- the-art results. However, those methods are hardly generalized to other ’low-resource’ instruments (such as guitar, cello, clarinet, etc.) transcription. In this paper, we propose a hybrid-supervised framework, the masked frame-level autoencoder (MFAE), to solve this issue. The proposed MFAE reconstructs the frame-level features of low-resource data to understand generic representations of low-resource instruments and improves low-resource transcription performance. Experimental results on several low- resource datasets (MAPS, MusicNet, and Guitarset) show that our framework achieves state-of-the-art performance in note-wise scores (Note F1 83.4%\64.1%\86.7%, Note-with-offset F1 59.8%\41.4%\71.6%). Moreover, our framework can be well generalized to various genres of instrument transcription, both in data-plentiful and data-limited scenarios.

关键词：

来源：评论

学校读者我要写书评

暂无评论

A Scalable Sparse Transformer Model for Singing Melody Extraction

A Scalable Sparse Transformer Model for Singing Melody Extra...

引用

International Conference on Acoustics, Speech, and Signal processing (ICASSP)

作者： Shuai Yu Jun Liu Yi Yu Wei Li School of Computer Science and Technology Donghua University Shanghai China National Institute of Informatics (NII) Tokyo Japan School of Computer Science and Technology Fudan University Shanghai China Shanghai Key Laboratory of Intelligent Information Processing Fudan University Shanghai China

Extracting the melody of a singing voice is an essential task within the realm of music information retrieval (MIR). Recently, transformer based models have drawn great attention in the field of MIR. However, due to the expensive computation cost and extensive parameters, it is difficult to train and deploy a transformer-based model for practical singing melody extraction. In this paper, we propose a simple yet effective scalable sparse transformer for singing melody extraction. To be specific, we first propose to employ a sparse transformer to reduce computation cost and the amount of parameters. Then, we proposed to scale the self-attention region of the sparse transformer in the spectrogram to obtain more accurate performance. Moreover, we propose to combine a scalable sparse transformer (S 2 Former) with CNN-based model to extract global and local features in the spectrogram. The proposed scalable transformer model can achieve a better balance between a standard transformer and a sparse transformer. To better fuse the features from transformer and CNN, we further propose a transformer-CNN fusion (TCF) module to combine significant features from transformer and CNN. The proposed model obtains state-of-the-art results on several public datasets. The conducted experiments confirm the effectiveness of the model we proposed.

关键词：

来源：评论

学校读者我要写书评

暂无评论

SPTNET: Span-based Prompt Tuning for Video Grounding

SPTNET: Span-based Prompt Tuning for Video Grounding

引用

IEEE International Conference on Multimedia and Expo (ICME)

作者： Yiren Zhang Yuanwu Xu Mohan Chen Yuejie Zhang Rui Feng Shang Gao School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University School of Information Technology Deakin University

When a Pre-trained Language Model (PLM) is adopted in video grounding task, it usually acts as a text encoder without having its knowledge fully utilized. Also, there exists an inconsistency problem between the pre-training and downstream objectives. To solve the issues, we propose a new paradigm, named Span-based Prompt Tuning (SPTNet). It can convert the video grounding task into a cloze form. Specifically, a query is first changed into a form with mask token by a template, then the video and the query embeddings are integrated through a cross-modal transformer. The start and end points of the query matching time span are predicted with the embedding of the mask token. Experimental results on two public benchmarks ActivityNet Captions and Charades-STA show that our SPTNet achieves surpassing performance compared with state-of-the-art methods.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Conditional Video-Text Reconstruction Network with Cauchy Mask for Weakly Supervised Temporal Sentence Grounding

Conditional Video-Text Reconstruction Network with Cauchy Ma...

引用

IEEE International Conference on Multimedia and Expo (ICME)

作者： Jueqi Wei Yuanwu Xu Mohan Chen Yuejie Zhang Rui Feng Shang Gao School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University School of Information Technology Deakin University

Temporal sentence grounding aims to detect the target segment most related to a given query in an untrimmed video. To alleviate the expensive annotation cost for temporal labels, researchers paid more attention to weakly supervised setting. Prior studies neglected the utilization of video representation reconstruction, which led to an unbalanced alignment learning. Moreover, they used different strategies to generate proposals which ignored the temporal structure in a query. In this paper, we propose a novel Conditional Video-Text Reconstruction Network (CVTRN). It supports conditional reconstruction of video and text representation. Specifically, video and text features are fused to compute semantic alignment, which is the condition of reconstruction. A new mask strategy for mask conditioned sentence reconstruction is also devised. This strategy focuses more on boundary regions than the widely used Gaussian mask in previous methods. Experimental results on two public benchmark datasets show that our CVTRN outperforms the state-of-the-art methods.

关键词：

来源：评论

学校读者我要写书评

暂无评论

GEP-DL4Mol: A Novel Molecular Deep-learning Model Optimization Framework for Boosting Molecular Properties Prediction

GEP-DL4Mol: A Novel Molecular Deep-learning Model Optimizati...

引用

2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023

作者： Peng, Yuzhong Zhang, Hao Zhang, Ziqiao Lin, Yanmei Zhou, Shuigeng Qiao, Shaojie Nanning Normal University Guangxi Key Lab of Human-Machine Interaction and Intelligent Decision Nanning China Chinese Academy of Sciences Shenzhen Institute of Advanced Technology Shenzhen China Fudan University Shanghai Key Lab of Intelligent Information Processing Shanghai China Nanning Normal University Undergraduate College Nanning China Chengdu University of Information Technology School of Software Engineering Chengdu China

ISBN: (纸本)9798350337488

High-performance molecular property prediction is one of the essential problems in many research fields. Although Deep Learning models (DLMs) have the strong potential to predict molecular property, the search space of their neural structure and hyperparameters is too huge to enumerate all possibilities, which implies that one will be difficult to get a optimal configuration solution, or even build a DLM with poor performance. To tackle the aforementioned problem, we first abstractly map various types of supervised DLMs for chemical molecular property prediction under a unified molecular DLM conceptualization. Then, we develop a self-learning molecular DLMs construction and optimization framework, called GEP-DL4Mol, based on the unified supervised molecular DLM conceptualization and Gene Expression Programming. Experiments on eight benchmark datasets including 19 property prediction tasks were conducted to evaluate the performance of the proposed method. Experimental results show that the proposed method outperforms other methods. © 2023 IEEE.

关键词： deep learning deep neural network optimization gene expression programming hyper-parameter optimization molecular modeling Molecular property prediction

来源：评论

学校读者我要写书评

暂无评论

Unified Lexical Representation for Interpretable Visual-Language Alignment 38

Unified Lexical Representation for Interpretable Visual-Lang...

引用

38th Conference on Neural information processing Systems, NeurIPS 2024

作者： Li, Yifan Wang, Yikai Fu, Yanwei Ru, Dongyu Zhang, Zheng He, Tong Fudan University China Amazon Web Services United States School of Data Science in Fudan Fudan ISTBI-ZJNU Algorithm Center for Brain-inspired Intelligence Shanghai Key Lab of Intelligent Information Processing and Technology Innovation Center of Calligraphy and Painting Digital Generation Ministry of Culture and Tourism China

Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work. Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores. On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words. However, lexical representations are difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively. In this paper, we introduce LexVLA, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design. We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability. To avoid the false discovery, we propose an overuse penalty to refrain the lexical representation from falsely frequently activating meaningless words. We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on the modest multi-modal dataset and avoid intricate training configurations. On cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g., 1.1B data, including CC-12M). We conduct extensive experiments to analyze LexVLA. Codes are available at https://***/Clementine24/LexVLA. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Camouflaged Object Segmentation with Transformer 6th

Camouflaged Object Segmentation with Transformer

引用

6th International Conference on Cognitive Systems and Signal processing, ICCSIP 2021

作者： Wang, Haiwen Wang, Xinzhou Sun, Fuchun Song, Yixu State Key Lab on Intelligent Technology and Systems Department of Computer Science and Technology Tsinghua University Beijing China School of Electronic and Information Engineering Tongji University Shanghai China

ISBN: (纸本)9789811692468

The Vision Transformer (ViT) [6] directly applies a Transformer architecture to image classification and achieves an impressive result compared with convolutional networks. This paper presents a new ViT-base camouflaged object segmentation method, called COS Transformer, which aims to identify and segment objects concealed in a complex environment. The high intrinsic similarities between object and surrounding makes the task challenging than salient object detection. Most recent camouflaged object segmentation methods(e.g., EGNet [29], PraNet [10] and SINet [9]) adopt convolutional network with an encoder-decoder architecture and focused on increasing the receptive field, which is limited by the depth of the network. In camouflaged object segmentation (COS) task, the camouflage is mainly relied on contrast of the whole surrounding instead of the local information. We introduce transformer with global context awareness in this paper, for self-attention allowing COS Transformer to aggregate features globally even in the lowest layers. Specifically, the architecture is composed of a transformer-based encoder and a multi-layers feature aggregation refinement module. After training on the COD10K [9] dataset, COS Transformer attains excellent results compared to state-of-the-art convolutional networks, e.g. 11.7% improvement of E [8] on the COD10K contrasted to SINet. © 2022, Springer Nature Singapore Pte Ltd.

关键词： Convolution

来源：评论

学校读者我要写书评

暂无评论

From Node Interaction to Hop Interaction: New Effective and Scalable Graph Learning Paradigm

From Node Interaction to Hop Interaction: New Effective and ...

引用

Conference on computer Vision and Pattern Recognition (CVPR)

作者： Jie Chen Zilong Li Yin Zhu Junping Zhang Jian Pu Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Institute of Science and Technology for Brain-Inspired Intelligence Fudan University Shanghai China

Existing Graph Neural Networks (GNNs) follow the message-passing mechanism that conducts information interaction among nodes iteratively. While considerable progress has been made, such node interaction paradigms still have the following limitation. First, the scalability limitation precludes the broad application of GNNs in large-scale industrial settings since the node interaction among rapidly expanding neighbors incurs high computation and memory costs. Second, the over-smoothing problem restricts the discrimination ability of nodes, i.e., node representations of different classes will converge to indistinguishable after repeated node interactions. In this work, we propose a novel hop interaction paradigm to address these limitations simultaneously. The core idea is to convert the interaction target among nodes to pre-processed multi-hop features inside each node. We design a simple yet effective HopGNN framework that can easily utilize existing GNNs to achieve hop interaction. Furthermore, we propose a multi-task learning strategy with a self-supervised learning objective to enhance HopGNN. We conduct extensive experiments on 12 benchmark datasets in a wide range of domains, scales, and smoothness of graphs. Experimental results show that our methods achieve superior performance while maintaining high scalability and efficiency. The code is at https://***/JC-202/HopGNN.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Math word problem solving with explicit numerical values 59

Math word problem solving with explicit numerical values

引用

Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language processing, ACL-IJCNLP 2021

作者： Wu, Qinzhuo Zhang, Qi Wei, Zhongyu Huang, Xuanjing Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

ISBN: (纸本)9781954085527

In recent years, math word problem solving has received considerable attention and achieved promising results, but previous methods rarely take numerical values into consideration. Most methods treat the numerical values in the problems as number symbols, and ignore the prominent role of the numerical values in solving the problem. In this paper, we propose a novel approach called NumS2T, which enhances math word problem solving performance by explicitly incorporating numerical values into a sequence-to-tree network. In addition, a numerical properties prediction mechanism is used to capture the category and comparison information of numerals and measure their importance in global expressions. Experimental results on the Math23K and APE datasets demonstrate that our model achieves better performance than existing state-of-the-art models. © 2021 Association for Computational Linguistics

关键词： Numerical methods

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：