Knowledge-based Visual Question Answering (KB-VQA) expands traditional VQA by utilizing world knowledge from external sources when the image alone is insufficient to infer a correct answer. Existing methods face chall...
详细信息
Knowledge-based Visual Question Answering (KB-VQA) expands traditional VQA by utilizing world knowledge from external sources when the image alone is insufficient to infer a correct answer. Existing methods face challenges due to low recall rates, limiting the ability to gather essential information for accurate answers. While increasing the amount of retrieved knowledge entries can enhance recall, it often introduces irrelevant information, adversely impairing model performance. To overcome these challenges, we propose RK-VQA, which comprises two components: First, a zero-shot weighted hybrid knowledge retrieval method that integrates local and global visual features with textual features from image-question pairs, enhancing the quality of knowledge retrieval and improving recall rates. Second, a rational knowledge-aware fusion- in-decoder architecture enhances answer generation by focusing on rational knowledge and reducing the influence of irrelevant information. Specifically, we develop a rational module to extract rational features, subsequently utilized to prioritize pertinent information via a novel rational knowledge-aware attention mechanism. We evaluate our RK-VQA on the OK-VQA, which is the largest knowledge-based VQA dataset. The results demonstrate that RK-VQA achieves significant results, recording an accuracy of 64.11%, surpassing the previous best result by 2.03%.
Open-source code often suffers from mismatched or missing comments, leading to difficult code comprehension, and burdening software development and maintenance. In this paper, we design a novel code summarization mode...
详细信息
ISBN:
(数字)9783031282386
ISBN:
(纸本)9783031282379;9783031282386
Open-source code often suffers from mismatched or missing comments, leading to difficult code comprehension, and burdening software development and maintenance. In this paper, we design a novel code summarization model CodeFiD to address this laborious challenge. Inspired by retrieval-augmented methods for open-domain question answering, CodeFiD first retrieves a set of relevant comments from code collections for a given code, and then aggregates presentations of code and these comments to produce a natural language sentence that summarizes the code behaviors. Different from current code summarization works that focus on improving code representations, our model resorts to external knowledge to enhance code summarizing performance. Extensive experiments on public code collections demonstrate the effectiveness of CodeFiD by outperforming state-of-the-art counterparts across all programming languages.
Retrieval-augmented generation models offer many benefits over standalone language models: besides a textual answer to a given query they provide provenance items retrieved from an updateable knowledge base. However, ...
详细信息
ISBN:
(纸本)9781450394086
Retrieval-augmented generation models offer many benefits over standalone language models: besides a textual answer to a given query they provide provenance items retrieved from an updateable knowledge base. However, they are also more complex systems and need to handle long inputs. In this work, we introduce FiD-Light to strongly increase the efficiency of the state-of-the-art retrieval-augmented FiD model, while maintaining the same level of effectiveness. Our FiD-Light model constrains the information flow from the encoder (which encodes passages separately) to the decoder (using concatenated encoded representations). Furthermore, we adapt FiD-Light with re-ranking capabilities through textual source pointers, to improve the top-ranked provenance precision. Our experiments on a diverse set of seven knowledge intensive tasks (KILT) show FiD-Light consistently improves the Pareto frontier between query latency and effectiveness. FiD-Light with source pointing sets substantial new state-of-the-art results on six KILT tasks for combined text generation and provenance retrieval evaluation, while maintaining high efficiency.
暂无评论