检索结果-内蒙古大学图书馆

TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Kim, Dohyeong Oh, Songhwai The Department of Electrical and Computer Engineering and ASRI Seoul National University Seoul08826 Korea Republic of

As safety is of paramount importance in robotics, reinforcement learning that reflects safety, called safe RL, has been studied extensively. In safe RL, we aim to find a policy which maximizes the desired return while satisfying the defined safety constraints. There are various types of constraints, among which constraints on conditional value at risk (CVaR) effectively lower the probability of failures caused by high costs since CVaR is a conditional expectation obtained above a certain percentile. In this paper, we propose a trust region-based safe RL method with CVaR constraints, called TRC. We first derive the upper bound on CVaR and then approximate the upper bound in a differentiable form in a trust region. Using this approximation, a subproblem to get policy gradients is formulated, and policies are trained by iteratively solving the subproblem. TRC is evaluated through safe navigation tasks in simulations with various robots and a sim-to-real environment with a Jackal robot from Clearpath. Compared to other safe RL methods, the performance is improved by 1.93 times while the constraints are satisfied in all experiments. Copyright © 2023, The Authors. All rights reserved.

关键词： Reinforcement learning

Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Risk

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Kim, Dohyeong Oh, Songhwai The Department of Electrical and Computer Engineering and ASRI Seoul National University Seoul08826 Korea Republic of

This paper aims to solve a safe reinforcement learning (RL) problem with risk measure-based constraints. As risk measures, such as conditional value at risk (CVaR), focus on the tail distribution of cost signals, constraining risk measures can effectively prevent a failure in the worst case. An on-policy safe RL method, called TRC, deals with a CVaR-constrained RL problem using a trust region method and can generate policies with almost zero constraint violations with high returns. However, to achieve outstanding performance in complex environments and satisfy safety constraints quickly, RL methods are required to be sample efficient. To this end, we propose an off-policy safe RL method with CVaR constraints, called off-policy TRC. If off-policy data from replay buffers is directly used to train TRC, the estimation error caused by the distributional shift results in performance degradation. To resolve this issue, we propose novel surrogate functions, in which the effect of the distributional shift can be reduced, and introduce an adaptive trust-region constraint to ensure a policy not to deviate far from replay buffers. The proposed method has been evaluated in simulation and real-world environments and satisfied safety constraints within a few steps while achieving high returns even in complex robotic tasks. Copyright © 2023, The Authors. All rights reserved.

关键词： Reinforcement learning

Distributionally Robust Differential Dynamic Programming with Wasserstein Distance

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Hakobyan, Astghik Yang, Insoon Department of Electrical and Computer Engineering and ASRI Seoul National University Seoul08826 Korea Republic of

Differential dynamic programming (DDP) is a popular technique for solving nonlinear optimal control problems with locally quadratic approximations. However, existing DDP methods are not designed for stochastic systems with unknown disturbance distributions. To address this limitation, we propose a novel DDP method that approximately solves the Wasserstein distributionally robust control (WDRC) problem, where the true disturbance distribution is unknown but a disturbance sample dataset is given. Our approach aims to develop a practical and computationally efficient DDP solution. To achieve this, we use the Kantrovich duality principle to decompose the value function in a novel way and derive closed-form expressions of the distributionally robust control and worst-case distribution policies to be used in each iteration of our DDP algorithm. This characterization makes our method tractable and scalable without the need for numerically solving any minimax optimization problems. The superior out-of-sample performance and scalability of our algorithm are demonstrated through kinematic car navigation and coupled oscillator problems. Copyright © 2023, The Authors. All rights reserved.

关键词： Stochastic systems

WayIL: Image-based Indoor Localization with Wayfinding Maps

学校读者我要写书评

暂无评论

WayIL: Image-based Indoor Localization with Wayfinding Maps

IEEE International Conference on Robotics and Automation (ICRA)

作者： Obin Kwon Dongki Jung Youngji Kim Soohyun Ryu Suyong Yeon Songhwai Oh Donghwan Lee Department of Electrical and Computer Engineering and ASRI Seoul National University Seoul South Korea NAVER LABS Seongnam South Korea

ISBN: (数字)9798350384574

ISBN: (纸本)9798350384581

This paper tackles a localization problem in large-scale indoor environments with wayfinding maps. A wayfinding map abstractly portrays the environment, and humans can localize themselves based on the map. However, when it comes to using it for robot localization, large geometrical discrepancies between the wayfinding map and the real world make it hard to use conventional localization methods. Our objective is to estimate a robot pose within a wayfinding map, utilizing RGB images from perspective cameras. We introduce two different imagination modules which are inspired by how humans can comprehend and interpret their surroundings for localization purposes. These modules jointly learn how to effectively observe the first-person-view (FPV) world to interpret bird-eye-view (BEV) maps. Providing explicit guidance to the two imagination modules significantly improves the precision of the localization system. We demonstrate the effectiveness of the proposed approach using real-world datasets, which are collected from various large-scale crowded indoor environments. The experimental results show that, in 85% of scenarios, the proposed localization system can estimate its pose within 3m in large indoor spaces. Project Site: https://***/projects/WayIL/

关键词： Location awareness Accuracy Navigation Semantics Robot vision systems Robot localization Particle filters

Semantic Environment Atlas for Object-Goal Navigation

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

作者： Kim, Nuri Park, Jeongho Hong, Mineui Oh, Songhwai Department of Electrical and Computer Engineering and ASRI Seoul National University 1 Gwanak-ro Gwanak-gu Seoul08826 Korea Republic of

In this paper, we introduce the Semantic Environment Atlas (SEA), a novel mapping approach designed to enhance visual navigation capabilities of embodied agents. The SEA utilizes semantic graph maps that intricately delineate the relationships between places and objects, thereby enriching the navigational context. These maps are constructed from image observations and capture visual landmarks as sparsely encoded nodes within the environment. The SEA integrates multiple semantic maps from various environments, retaining a memory of place-object relationships, which proves invaluable for tasks such as visual localization and navigation. We developed navigation frameworks that effectively leverage the SEA, and we evaluated these frameworks through visual localization and object-goal navigation tasks. Our SEA-based localization framework significantly outperforms existing methods, accurately identifying locations from single query images. Experimental results in Habitat scenarios show that our method not only achieves a success rate of 39.0%—an improvement of 12.4% over the current state-of-the-art—but also maintains robustness under noisy odometry and actuation conditions, all while keeping computational costs low. © 2024, The Authors. All rights reserved.

关键词： Semantics

Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approach

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Kim, Dohyeong Kwon, Hyeokjin Kim, Junseok Lee, Gunmin Oh, Songhwai The Department of Electrical and Computer Engineering and ASRI Seoul National University Seoul08826 Korea Republic of The Interdisciplinary Program in Artificial Intelligence and ASRI Seoul National University Seoul08826 Korea Republic of

As the complexity of tasks addressed through reinforcement learning (RL) increases, the definition of reward functions also has become highly complicated. We introduce an RL method aimed at simplifying the reward-shaping process through intuitive strategies. Initially, instead of a single reward function composed of various terms, we define multiple reward and cost functions within a constrained multi-objective RL (CMORL) framework. For tasks involving sequential complex movements, we segment the task into distinct stages and define multiple rewards and costs for each stage. Finally, we introduce a practical CMORL algorithm that maximizes objectives based on these rewards while satisfying constraints defined by the costs. The proposed method has been successfully demonstrated across a variety of acrobatic tasks in both simulation and real-world environments. Additionally, it has been shown to successfully perform tasks compared to existing RL and constrained RL algorithms. Our code is available at https://***/rllab-snu/Stage-Wise-CMORL. Copyright © 2024, The Authors. All rights reserved.

关键词： Cost functions

Semantic Environment Atlas for Object-Goal Navigation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Kim, Nuri Park, Jeongho Hong, Mineui Oh, Songhwai Department of Electrical and Computer Engineering and ASRI Seoul National University 1 Gwanak-ro Gwanak-gu Seoul08826 Korea Republic of

In this paper, we introduce the Semantic Environment Atlas (SEA), a novel mapping approach designed to enhance visual navigation capabilities of embodied agents. The SEA utilizes semantic graph maps that intricately delineate the relationships between places and objects, thereby enriching the navigational context. These maps are constructed from image observations and capture visual landmarks as sparsely encoded nodes within the environment. The SEA integrates multiple semantic maps from various environments, retaining a memory of place-object relationships, which proves invaluable for tasks such as visual localization and navigation. We developed navigation frameworks that effectively leverage the SEA, and we evaluated these frameworks through visual localization and object-goal navigation tasks. Our SEA-based localization framework significantly outperforms existing methods, accurately identifying locations from single query images. Experimental results in Habitat [1] scenarios show that our method not only achieves a success rate of 39.0%—an improvement of 12.4% over the current state-of-the-art—but also maintains robustness under noisy odometry and actuation conditions, all while keeping computational costs low. © 2024, CC BY.

关键词： Mapping

EdiText: Controllable Coarse-to-Fine Text Editing with Diffusion Language Models

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Lee, Che Hyun Kim, Heeseung Yeom, Jiheum Yoon, Sungroh Department of Electrical and Computer Engineering Seoul National University Korea Republic of AIIS ASRI INMC ISRC IPAI Seoul National University Korea Republic of

We propose EdiText, a controllable text editing method that modify the reference text to desired attributes at various scales. We integrate an SDEdit-based editing technique that allows for broad adjustments in the degree of text editing. Additionally, we introduce a novel fine-level editing method based on self-conditioning, which allows subtle control of reference text. While being capable of editing on its own, this fine-grained method, integrated with the SDEdit approach, enables EdiText to make precise adjustments within the desired range. EdiText demonstrates its controllability to robustly adjust reference text at broad range of levels across various tasks, including toxicity control and sentiment control. © 2025, CC BY.

关键词：

Listwise Reward Estimation for Offline Preference-based Reinforcement Learning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Choi, Heewoong Jung, Sangwon Ahn, Hongjoon Moon, Taesup Department of Electrical and Computer Engineering Seoul National University Korea Republic of ASRI INMC IPAI AIIS Seoul National University Korea Republic of

In Reinforcement Learning (RL), designing precise reward functions remains to be a challenge, particularly when aligning with human intent. Preference-based RL (PbRL) was introduced to address this problem by learning reward models from human feedback. However, existing PbRL methods have limitations as they often overlook the second-order preference that indicates the relative strength of preference. In this paper, we propose Listwise Reward Estimation (LiRE), a novel approach for offline PbRL that leverages second-order preference information by constructing a Ranked List of Trajectories (RLT), which can be efficiently built by using the same ternary feedback type as traditional methods. To validate the effectiveness of LiRE, we propose a new offline PbRL dataset that objectively reflects the effect of the estimated rewards. Our extensive experiments on the dataset demonstrate the superiority of LiRE, i.e., outperforming state-of-the-art baselines even with modest feedback budgets and enjoying robustness with respect to the number of feedbacks and feedback noise. Our code is available at https://***/chwoong/LiRE Copyright © 2024, The Authors. All rights reserved.

关键词： Reinforcement learning