检索结果-内蒙古大学图书馆

ESceme: vision-and-language navigation with Episodic Scene Memory

INTERNATIONAL JOURNAL OF COMPUTER vision 2025年第1期133卷 254-274页

作者： Zheng, Qi Liu, Daqing Wang, Chaoyue Zhang, Jing Wang, Dadong Tao, Dacheng Shenzhen Univ Coll Elect & Informat Engn Shenzhen 518000 Peoples R China Univ Sydney Sch Comp Sci Sydney NSW 2008 Australia JD Explore Acad Beijing Peoples R China CSIRO Data 61 Sydney NSW 2122 Australia

vision-and-language navigation (VLN) simulates a visual agent that follows natural-language navigation instructions in real-world scenes. Existing approaches have made enormous progress in navigation in new environments, such as beam search, pre-exploration, and dynamic or hierarchical history encoding. To balance generalization and efficiency, we resort to memorizing visited scenarios apart from the ongoing route while navigating. In this work, we introduce a mechanism of Episodic Scene memory (ESceme) for VLN that wakes an agent's memories of past visits when it enters the current scene. The episodic scene memory allows the agent to envision a bigger picture of the next prediction. This way, the agent learns to utilize dynamically updated information instead of merely adapting to the current observations. We provide a simple yet effective implementation of ESceme by enhancing the accessible views at each location and progressively completing the memory while navigating. We verify the superiority of ESceme on short-horizon (R2R), long-horizon (R4R), and vision-and-dialog (CVDN) VLN tasks. Our ESceme also wins first place on the CVDN leaderboard. Code is available: https://***/qizhust/esceme.

关键词： vision-and-language navigation Embodied AI Scene memory Episodic memory Multimodal fusion

来源：评论

学校读者我要写书评

暂无评论

Hijacking vision-and-language navigation Agents with Adversarial Environmental Attacks

Hijacking Vision-and-Language Navigation Agents with Adversa...

引用

2025 IEEE/CVF Winter Conference on Applications of Computer vision, WACV 2025

作者： Yang, Zijiao Shi, Xiangxi Slyman, Eric Lee, Stefan Oregon State University United States

ISBN: (纸本)9798331510831

Assistive embodied agents that can be instructed in natural language to perform tasks in open-world environments have the potential to significantly impact labor tasks like manufacturing or in-home care - benefiting the lives of those who come to depend on them. In this work, we consider how this benefit might be hijacked by local modifications in the appearance of the agent's operating environment. Specifically, we take the popular vision-and-language navigation (VLN) task as a representative setting and develop a whitebox adversarial attack that optimizes a 3D attack object's appearance to induce desired behaviors in pretrained VLN agents that observe it in the environment. We demonstrate that the proposed attack can cause VLN agents to ignore their instructions and execute alternative actions after encountering the attack object - even for instructions and agent paths not considered when optimizing the attack. For these novel settings, we find our attacks can induce early-termination behaviors or divert an agent along an attacker-defined multi-step trajectory. Under both conditions, environmental attacks significantly reduce agent capabilities to successfully follow user instructions. © 2025 IEEE.

关键词： adversarial attack vision-and-language navigation

来源：评论

学校读者我要写书评

暂无评论

Recursive bidirectional cross-modal reasoning network for vision-and-language navigation

引用

EXPERT SYSTEMS WITH APPLICATIONS 2025年 270卷

作者： Wu, Jie Wu, Chunlei Shen, Xiuxuan Wu, Fengjiang Wang, Leiquan China Univ Petr East China Qingdao Inst Software Coll Comp Sci & Technol Qingdao Peoples R China Xidian Univ Sch Cyber Engn Xian Peoples R China

vision-and-language navigation (VLN) requires an agent to comprehend natural language instructions and autonomously navigate in unknown environments to reach specified target locations. Traditional approaches primarily employ graph networks or transformers to model the relationships between different modalities. However, they often struggle with catastrophic forgetting of historical information and information loss between layers in multi-layer architectures, which leads to a confusion for the navigation decision. To address these challenges, we propose the Recursive Bidirectional Cross-Modal Reasoning Network (RBCRN) to continuously explore the correspondences between observations and instructions while avoiding the forgetting of historical information. Additionally, the Recursive Memory Integration Unit (RMIU) is designed to preserve and transmit crucial information by combining the output of each layer with the memory state of the previous layer, thereby reducing information loss between multiple layers and improving the overall utilization of contextual information. Comprehensive experiments on the R2R, SOON, and REVERIE datasets demonstrate that our approach significantly outperforms existing methods in VLN.

关键词： Memory iteration Long-term dependency vision-and-language navigation

来源：评论

学校读者我要写书评

暂无评论

DynamicVLN: Incorporating Dynamics into vision-and-language navigation Scenarios

引用

SENSORS 2025年第2期25卷 364页

作者： Sun, Yanjun Qiu, Yue Aoki, Yoshimitsu Keio Univ Dept Elect & Elect Engn Fac Sci & Technol 3-14-1 HiyoshiKohoku Ku Yokohama 2238522 Japan Natl Inst Adv Ind Sci & Technol 1-1-1 Umezono Tsukuba 3058560 Japan

Traditional vision-and-language navigation (VLN) tasks require an agent to navigate static environments using natural language instructions. However, real-world road conditions such as vehicle movements, traffic signal fluctuations, pedestrian activity, and weather variations are dynamic and continually changing. These factors significantly impact an agent's decision-making ability, underscoring the limitations of current VLN models, which do not accurately reflect the complexities of real-world navigation. To bridge this gap, we propose a novel task called Dynamic vision-and-language navigation (DynamicVLN), incorporating various dynamic scenarios to enhance the agent's decision-making abilities and adaptability. By redefining the VLN task, we emphasize that a robust and generalizable agent should not rely solely on predefined instructions but must also demonstrate reasoning skills and adaptability to unforeseen events. Specifically, we have designed ten scenarios that simulate the challenges of dynamic navigation and developed a dedicated dataset of 11,261 instances using the CARLA simulator (ver.0.9.13) and large language model to provide realistic training conditions. Additionally, we introduce a baseline model that integrates advanced perception and decision-making modules, enabling effective navigation and interpretation of the complexities of dynamic road conditions. This model showcases the ability to follow natural language instructions while dynamically adapting to environmental cues. Our approach establishes a benchmark for developing agents capable of functioning in real-world, dynamic environments and extending beyond the limitations of static VLN tasks to more practical and versatile applications.

关键词： vision-and-language navigation dynamic change decision-making

来源：评论

学校读者我要写书评

暂无评论

vision-and-language navigation via Latent Semantic Alignment Learning

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2024年 26卷 8406-8418页

作者： Wu, Siying Fu, Xueyang Wu, Feng Zha, Zheng-Jun Hefei Comprehens Natl Sci Ctr Inst Artificial Intelligence Hefei 230039 Peoples R China Univ Sci & Technol China Sch Informat Sci & Technol Hefei 230026 Peoples R China

vision-and-language navigation (VLN) requires that an agent can comprehensively understand the given instructions and the immediate visual information obtained from the environment, so as to make correct actions to achieve the navigation goal. Therefore, semantic alignment across modalities is crucial for the agent understanding its own state during the navigation process. However, the potential of semantic alignment has not been systematically explored in current studies, which limits the further improvement of navigation performance. To address this issue, we propose a new Latent Semantic Alignment Learning method to develop the semantically aligned relationships contained in the environment. Specifically, we introduce three novel pre-training tasks: Trajectory-conditioned Masked Fragment Modeling, Action Prediction of Masked Observation, and Hierarchical Triple Contrastive Learning. The first two tasks are used to reason about cross-modal dependencies, while the third one is able to learn semantically consistent representations across modalities. In this way, the Latent Semantic Alignment Learning method establishes a consistent perception of the environment and makes the agent's actions easier to explain. Experiments on common benchmarks verify the effectiveness of our proposed methods. For example, we improve the Success Rate by 1.6% on the R2R validation unseen set and 4.3% on the R4R validation unseen set over the baseline model.

关键词： vision-and-language navigation visual-language pre-training semantic alignment vision-and-language navigation visual-language pre-training semantic alignment

来源：评论

学校读者我要写书评

暂无评论

vision-and-language navigation via Causal Learning

Vision-and-Language Navigation via Causal Learning

引用

IEEE/CVF Conference on Computer vision and Pattern Recognition (CVPR)

作者： Wang, Liuyi He, Zongtao Dang, Ronghao Shen, Mengjiao Liu, Chengju Chen, Qijun Tongji Univ Sch Elect & Informat Engn Shanghai Peoples R China

ISBN: (纸本)9798350353006

In the pursuit of robust and generalizable environment perception and language understanding, the ubiquitous challenge of dataset bias continues to plague vision-and-language navigation (VLN) agents, hindering their perfor-mance in unseen environments. This paper introduces the generalized cross-modal causal transformer (GOAT), a pioneering solution rooted in the paradigm of causal inference. By delving into both observable and unobservable confounders within vision, language, and history, we propose the back-door and front-door adjustment causal learning (BACL and FACL) modules to promote unbiased learning by comprehensively mitigating potential spurious correlations. Additionally, to capture global confounder features, we propose a cross-modal feature pooling (CFP) module supervised by contrastive learning, which is also shown to be effective in improving cross-modal representations during pre-training. Extensive experiments across multiple VLN datasets (R2R, REVERIE, RxR, and SOON) under-score the superiority of our proposed method over previous state-of-the-art approaches. Code is available at https://***/CrystalSixone/VLN-GOAT.

关键词： causal learning cross-modal fusion embodied AI vision-and-language vision-and-language navigation

来源：评论

学校读者我要写书评

暂无评论

Adaptive Cross-Modal Experts Network with Uncertainty-Driven Fusion for vision-language navigation

引用

KNOWLEDGE-BASED SYSTEMS 2025年 307卷

作者： Wu, Jie Wu, Chunlei Shen, Xiuxuan Wang, Leiquan China Univ Petr East China Qingdao Inst Software Coll Comp Sci & Technol Qingdao Peoples R China Xidian Univ Sch Cyber Engn Xian Peoples R China

vision-and-language navigation (VLN) enables an agent to autonomously navigate in real-world environments based on language instructions to reach specified destinations and accurately locate relevant targets. Although significant progress has been made in recent years, two major limitations remain: (1) Existing methods lack flexibility and diversity in processing multimodal information and cannot dynamically adjust to different input features. (2) Current fixed fusion strategies fail to dynamically adapt to varying data quality in open environments, insufficiently leveraging multi-scale features and handling complex nonlinear relationships. In this paper, an adaptive cross-modal experts network (ACME) with uncertainty-driven fusion is proposed to address these issues. The adaptive cross-modal experts module dynamically selects the most suitable expert network based on the input features, enhancing information processing diversity and flexibility. Additionally, the uncertainty-driven fusion module balances coarse-grained and fine-grained information by calculating their confidences and dynamically adjusting the fusion weights. Comprehensive experiments on the R2R, SOON, and REVERIE datasets demonstrate that our approach significantly outperforms existing VLN approaches.

关键词： vision-and-language navigation Uncertainty-driven fusion Mixture of experts

来源：评论

学校读者我要写书评

暂无评论

Boosting vision-and-language navigation with Direction Guiding and Backtracing

引用

ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS 2023年第1期19卷 1-16页

作者： Chen, Jingwen Luo, Jianjie Pan, Yingwei Li, Yehao Yao, Ting Chao, Hongyang Mei, Tao Sun Yat Sen Univ Guangzhou Higher Educ Mega Ctr 132 Waihuan East Rd Guangzhou 510006 Peoples R China JD AI Res 8 Beichen West St Beijing 100105 Peoples R China

vision-and-language navigation (VLN) has been an emerging and fast-developing research topic, where an embodied agent is required to navigate in a real-world environment based on natural language instructions. In this article, we present a Direction-guided Navigator Agent (DNA) that novelly integrates direction clues derived from instructions into the essential encoder-decoder navigation framework. Particularly, DNA couples the standard instruction encoder with an additional direction branch which sequentially encodes the direction clues in the instructions to boost navigation. Furthermore, an Instruction Flipping mechanism is uniquely devised to enable fast data augmentation as well as a follow-up backtracing for navigating the agent in a backward direction. Such a way naturally amplifies the grounding of instruction in the local visual scenes along both forward and backward directions, and thus strengthens the alignment between instruction and action sequence. Extensive experiments conducted on Room to Room (R2R) dataset validate our proposal and demonstrate quantitatively compelling results.

关键词： vision-and-language navigation cross-modal matching

来源：评论

学校读者我要写书评

暂无评论

Memory-Adaptive vision-and-language navigation

引用

PATTERN RECOGNITION 2024年 153卷

作者： He, Keji Jing, Ya Huang, Yan Lu, Zhihe An, Dong Wang, Liang Chinese Acad Sci Inst Automat Ctr Res Intelligent Percept & Comp State Key Lab Multimodal Artificial Intelligence S Beijing Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing Peoples R China ByteDance AI Lab Beijing Peoples R China Natl Univ Singapore Singapore Singapore Univ Chinese Acad Sci Sch Future Technol Beijing Peoples R China

vision -and -language navigation (VLN) requests an agent to navigate in 3D environments following given instructions, where history is critical for decision -making in dynamic navigation process. Particularly, a memory bank storing histories is widely used in existing methods to incorporate with multimodel representations in current scenes for better decision -making. However, by weighting each history with a simple scalar, those methods cannot purely utilize the informative cues that co -exist with detrimental contents in each history, thereby inevitably introducing noises into decision -making. To that end, we propose a novel Memory -Adaptive Model (MAM) that can dynamically restrain the detrimental contents in histories for retaining contents that benefit navigation only. Specifically, two key modules, Visual and Textual Adaptive Modules, are designed to restrain history noises based on scene -related vision and text, respectively. A Reliability Estimator Module is further introduced to refine above adaptation operations. Our experiments on the widely used RxR and R2R datasets show that MAM outperforms its baseline method by 4.0% / 2.5% and 2% / 1% on the validation unseen/test split, respectively, wrt the SR metric.

关键词： vision-and-language navigation Memory bank History noises Memory-Adaptive Model

来源：评论

学校读者我要写书评

暂无评论

Talk2Nav: Long-Range vision-and-language navigation with Dual Attention and Spatial Memory

引用

INTERNATIONAL JOURNAL OF COMPUTER vision 2021年第1期129卷 246-266页

作者： Vasudevan, Arun Balajee Dai, Dengxin Van Gool, Luc Swiss Fed Inst Technol Zurich Switzerland Katholieke Univ Leuven Leuven Belgium

The role of robots in society keeps expanding, bringing with it the necessity of interacting and communicating with humans. In order to keep such interaction intuitive, we provide automatic wayfinding based on verbal navigational instructions. Our first contribution is the creation of a large-scale dataset with verbal navigation instructions. To this end, we have developed an interactive visual navigation environment based on Google Street View;we further design an annotation method to highlight mined anchor landmarks and local directions between them in order to help annotators formulate typical, human references to those. The annotation task was crowdsourced on the AMT platform, to construct a new Talk2Nav dataset with 10, 714 routes. Our second contribution is a new learning method. Inspired by spatial cognition research on the mental conceptualization of navigational instructions, we introduce a soft dual attention mechanism defined over the segmented language instructions to jointly extract two partial instructions-one for matching the next upcoming visual landmark and the other for matching the local directions to the next landmark. On the similar lines, we also introduce spatial memory scheme to encode the local directional transitions. Our work takes advantage of the advance in two lines of research: mental formalization of verbal navigational instructions and training neural network agents for automatic way finding. Extensive experiments show that our method significantly outperforms previous navigation methods. For demo video, dataset and code, please refer to our project page..

关键词： vision-and-language navigation Long-range navigation Spatial memory Dual attention

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：