检索结果-内蒙古大学图书馆

4th Annual Conference on learning for Dynamics and Control (L4DC)

作者： Cosner, Ryan K. Tucker, Maegan Taylor, Andrew J. Li, Kejun Molnar, Tamas G. Ubellacker, Wyatt Alan, Anil Orosz, Gabor Yue, Yisong Ames, Aaron D. CALTECH Pasadena CA 91125 USA Univ Michigan Ann Arbor MI 48109 USA Argo AI Pittsburgh PA USA

Bringing dynamic robots into the wild requires a tenuous balance between performance and safety. Yet controllers designed to provide robust safety guarantees often result in conservative behavior, and tuning these controllers to find the ideal trade-off between performance and safety typically requires domain expertise or a carefully constructed reward function. This work presents a design paradigm for systematically achieving behaviors that balance performance and robust safety by integrating safety-aware preference-based learning (PBL) with Control Barrier Functions (CBFs). Fusing these concepts-safety-aware learning and safety-critical control-gives a robust means to achieve safe behaviors on complex robotic systems in practice. We demonstrate the capability of this design paradigm to achieve safe and performant perception-based autonomous operation of a quadrupedal robot both in simulation and experimentally on hardware.

关键词： preference-based learning Control Barrier Functions Safety-Critical Control Robotics

来源：评论

学校读者我要写书评

暂无评论

APReL: A Library for Active preference-based Reward learning Algorithms 22

APReL: A Library for Active Preference-based Reward Learning...

引用

17th Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI)

作者： Biyik, Erdem Talati, Aditi Sadigh, Dorsa Stanford Univ Elect Engn Stanford CA 94305 USA Stanford Univ Comp Sci Stanford CA USA Stanford Univ Comp Sci & Elect Engn Stanford CA USA

ISBN: (数字)9781665407311

ISBN: (纸本)9781665407311

Reward learning is a fundamental problem in human-robot interaction to have robots that operate in alignment with what their human user wants. Many preference-based learning algorithms and active querying techniques have been proposed as a solution to this problem. In this paper, we present APReL, a library for active preference-based reward learning algorithms, which enable researchers and practitioners to experiment with the existing techniques and easily develop their own algorithms for various modules of the problem. APReL is available at https://***/Stanford-ILIAD/APReL.

关键词： reward learning active learning software library preference-based learning

来源：评论

学校读者我要写书评

暂无评论

Active preference-based Gaussian process regression for reward learning and optimization

引用

INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH 2024年第5期43卷 665-684页

作者： Biyik, Erdem Huynh, Nicolas Kochenderfer, Mykel J. Sadigh, Dorsa Stanford Univ Dept Elect Engn Stanford CA USA Univ Calif Berkeley Ctr Human Compatible Artificial Intelligence Berkeley CA USA Univ Southern Calif Thomas Lord Dept Comp Sci Los Angeles CA USA Ecole Polytech Dept Appl Math Palaiseau France Univ Cambridge Dept Comp Sci & Technol Cambridge England Stanford Univ Dept Aeronaut & Astronaut Stanford CA USA Stanford Univ Dept Comp Sci Stanford CA USA Univ Southern Calif 3737 Watt WayPowell Hall PHERoom 214 Los Angeles CA 90089 USA

Designing reward functions is a difficult task in AI and robotics. The complex task of directly specifying all the desirable behaviors a robot needs to optimize often proves challenging for humans. A popular solution is to learn reward functions using expert demonstrations. This approach, however, is fraught with many challenges. Some methods require heavily structured models, for example, reward functions that are linear in some predefined set of features, while others adopt less structured reward functions that may necessitate tremendous amounts of data. Moreover, it is difficult for humans to provide demonstrations on robots with high degrees of freedom, or even quantifying reward values for given trajectories. To address these challenges, we present a preference-based learning approach, where human feedback is in the form of comparisons between trajectories. We do not assume highly constrained structures on the reward function. Instead, we employ a Gaussian process to model the reward function and propose a mathematical formulation to actively fit the model using only human preferences. Our approach enables us to tackle both inflexibility and data-inefficiency problems within a preference-based learning framework. We further analyze our algorithm in comparison to several baselines on reward optimization, where the goal is to find the optimal robot trajectory in a data-efficient way instead of learning the reward function for every possible trajectory. Our results in three different simulation experiments and a user study show our approach can efficiently learn expressive reward functions for robotic tasks, and outperform the baselines in both reward learning and reward optimization.

关键词： Reward learning active learning inverse reinforcement learning preference-based learning human-robot interaction Gaussian processes

来源：评论

学校读者我要写书评

暂无评论

preference learning for guiding the tree search in continuous POMDPs 7

Preference learning for guiding the tree search in continuou...

引用

7th Conference on Robot learning (CoRL)

作者： Ahn, Jiyong Son, Sanghyeon Lee, Dongryung Han, Jisu Son, Dongwon Kim, Beomjoon Korea Adv Inst Sci & Technol Grad Sch AI Daejeon South Korea

A robot operating in a partially observable environment must perform sensing actions to achieve a goal, such as clearing the objects in front of a shelf to better localize a target object at the back, and estimate its shape for grasping. A POMDP is a principled framework for enabling robots to perform such information-gathering actions. Unfortunately, while robot manipulation domains involve high-dimensional and continuous observation and action spaces, most POMDP solvers are limited to discrete spaces. Recently, POMCPOW [1] has been proposed for continuous POMDPs, which handles continuity using sampling and progressive widening [2]. However, for robot manipulation problems involving camera observations and multiple objects, POMCPOW is too slow to be practical. We take inspiration from the recent work in learning to guide task and motion planning [3] to propose a framework that learns to guide POMCPOW from past planning experience. Our method uses preference learning [4, 5, 6, 7] that utilizes both success and failure trajectories, where the preference label is given by the results of the tree search. We demonstrate the efficacy of our framework in several continuous partially observable robotics domains, including real-world manipulation, where our framework explicitly reasons about the uncertainty in off-the-shelf segmentation and pose estimation algorithms. Details of the project are accessible in the following URL: https://***/view/preference-guided-pomcpow?usp=sharing.

关键词： POMDP Online planning Guided Search preference-based learning

来源：评论

学校读者我要写书评

暂无评论

Incentivizing Efficient Equilibria in Traffic Networks With Mixed Autonomy

引用

IEEE TRANSACTIONS ON CONTROL OF NETWORK SYSTEMS 2021年第4期8卷 1717-1729页

作者： Biyik, Erdem Lazar, Daniel A. Pedarsani, Ramtin Sadigh, Dorsa Stanford Univ Dept Elect Engn Stanford CA 94305 USA UC Santa Barbara Dept Elect & Comp Engn Santa Barbara CA 93106 USA Stanford Univ Dept Comp Sci Stanford CA 94305 USA

Traffic congestion has large economic and social costs. The introduction of autonomous vehicles can potentially reduce this congestion by increasing road capacity via vehicle platooning and by creating an avenue for influencing people's choice of routes. We consider a network of parallel roads with two modes of transportation: 1) human drivers, who will choose the quickest route available to them, and 2) a ride hailing service, which provides an array of autonomous vehicle route options, each with different prices, to users. We formalize a model of vehicle flow in mixed autonomy and a model of how autonomous service users make choices between routes with different prices and latencies. Developing an algorithm to learn the preferences of the users, we formulate a planning optimization that chooses prices to maximize a social objective. We demonstrate the benefit of the proposed scheme by comparing the results to theoretical benchmarks that, we show, can be efficiently calculated.

关键词： Mixed-autonomy traffic preference-based learning vehicle routing

来源：评论

学校读者我要写书评

暂无评论

learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences

引用

INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH 2022年第1期41卷 45-67页

作者： Biyik, Erdem Losey, Dylan P. Palan, Malayandi Landolfi, Nicholas C. Shevchuk, Gleb Sadigh, Dorsa Stanford Univ Dept Elect Engn 353 Jane Stanford Way Stanford CA 94305 USA Stanford Univ Dept Comp Sci Stanford CA 94305 USA

Reward functions are a common way to specify the objective of a robot. As designing reward functions can be extremely challenging, a more promising approach is to directly learn reward functions from human teachers. Importantly, data from human teachers can be collected either passively or actively in a variety of forms: passive data sources include demonstrations (e.g., kinesthetic guidance), whereas preferences (e.g., comparative rankings) are actively elicited. Prior research has independently applied reward learning to these different data sources. However, there exist many domains where multiple sources are complementary and expressive. Motivated by this general problem, we present a framework to integrate multiple sources of information, which are either passively or actively collected from human users. In particular, we present an algorithm that first utilizes user demonstrations to initialize a belief about the reward function, and then actively probes the user with preference queries to zero-in on their true reward. This algorithm not only enables us combine multiple data sources, but it also informs the robot when it should leverage each type of information. Further, our approach accounts for the human's ability to provide data: yielding user-friendly preference queries which are also theoretically optimal. Our extensive simulated experiments and user studies on a Fetch mobile manipulator demonstrate the superiority and the usability of our integrated framework.

关键词： Reward learning active learning inverse reinforcement learning learning from demonstrations preference-based learning human-robot interaction

来源：评论

学校读者我要写书评

暂无评论

Hierarchical learning from human preferences and curiosity

引用

APPLIED INTELLIGENCE 2022年第7期52卷 7459-7479页

作者： Bougie, Nicolas Ichise, Ryutaro Grad Univ Adv Studies Sokendai Tokyo Japan Natl Inst Informat Tokyo Japan

Recent success in scaling deep reinforcement algorithms (DRL) to complex problems has been driven by well-designed extrinsic rewards, which limits their applicability to many real-world tasks where rewards are naturally extremely sparse. One solution to this problem is to introduce human guidance to drive the agent's learning. Although low-level demonstrations is a promising approach, it was shown that such guidance may be difficult for experts to demonstrate since some tasks require a large amount of high-quality demonstrations. In this work, we explore human guidance in the form of high-level preferences between sub-goals, leading to drastic reductions in both human effort and cost of exploration. We design a novel hierarchical reinforcement learning method that introduces non-expert human preferences at the high-level, and curiosity to drastically speed up the convergence of subpolicies to reach any sub-goals. We further propose a strategy based on curiosity to automatically discover sub-goals. We evaluate the proposed method on 2D navigation tasks, robotic control tasks, and image-based video games (Atari 2600), which have high-dimensional observations, sparse rewards, and complex state dynamics. The experimental results show that the proposed method can learn significantly faster than traditional hierarchical RL methods and drastically reduces the amount of human effort required over standard imitation learning approaches.

关键词： Hierarchical reinforcement learning preference-based learning Curiosity Human guidance

来源：评论

学校读者我要写书评

暂无评论

Batch Active learning of Reward Functions from Human preferences

引用

ACM TRANSACTIONS ON HUMAN-ROBOT INTERACTION 2024年第2期13卷 1-27页

作者： Biyik, Erdem Anari, Nima Sadigh, Dorsa Univ Southern Calif Thomas Lord Dept Comp Sci 3737 Watt WayPowell Hall PHE Los Angeles CA 90089 USA Stanford Univ Dept Comp Sci Gates Comp Sci 168A353 Jane Stanford Way Stanford CA 94305 USA

Data generation and labeling are often expensive in robot learning. preference-based learning is a concept that enables reliable labeling by querying users with preference questions. Active querying methods are commonly employed in preference-based learning to generate more informative data at the expense of parallelization and computation time. In this article, we develop a set of novel algorithms, batch active preference-based learning methods, that enable efficient learning of reward functions using as few data samples as possible while still having short query generation times and also retaining parallelizability. We introduce a method based on determinantal point processes for active batch generation and several heuristic-based alternatives. Finally, we present our experimental results for a variety of robotics tasks in simulation. Our results suggest that our batch active learning algorithm requires only a few queries that are computed in a short amount of time. We showcase one of our algorithms in a study to learn human users' preferences.

关键词： Reward learning active learning preference-based learning human-robot interaction robot learning inverse reinforcement learning

来源：评论

学校读者我要写书评

暂无评论

Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizability 33

Efficient and Optimal Algorithms for Contextual Dueling Band...

引用

33rd International Conference on Algorithmic learning Theory (ALT)

作者： Saha, Aadirupa Krishnamurthy, Akshay Microsoft Res New York NY 10012 USA

We study the K-armed contextual dueling bandit problem, a sequential decision making setting in which the learner uses contextual information to make two decisions, but only observes preference-based feedback suggesting that one decision was better than the other. We focus on the regret minimization problem under realizability, where the feedback is generated by a pairwise preference matrix that is well-specified by a given function class F. We provide a new algorithm that achieves the optimal regret rate for a new notion of best response regret, which is a strictly stronger performance measure than those considered in prior works. The algorithm is also computationally efficient, running in polynomial time assuming access to an online oracle for square loss regression over F. This resolves an open problem of Dudik et al. (2015) on oracle efficient, regret-optimal algorithms for contextual dueling bandits.

关键词： Contextual Dueling Bandits preference-based learning Realizability Function approximation Regret analysis Best-Response Policy regret Regression oracles Efficient Optimal algorithms Markov games Linear realizability Agnostic

来源：评论

学校读者我要写书评

暂无评论

preferences-based Choice Prediction in Evolutionary Multi-objective Optimization 20th

Preferences-Based Choice Prediction in Evolutionary Multi-ob...

引用

20th European Conference on the Applications of Evolutionary Computation (EvoApplications)

作者： Aggarwal, Manish Heinermann, Justin Oehmcke, Stefan Kramer, Oliver Indian Inst Management Ahmedabad Dept Informat Syst Ahmadabad Gujarat India Carl von Ossietzky Univ Oldenburg Dept Comp Sci Computat Intelligence Grp Oldenburg Germany

ISBN: (纸本)9783319558493;9783319558486

Evolutionary multi-objective algorithms (EMOAs) of the type of NSGA-2 approximate the Pareto-front, after which a decision-maker (DM) is confounded with the primary task of selecting the best solution amongst all the equally good solutions on the Pareto-front. In this paper, we complement the popular NSGA-2 EMOA by posteriori identifying a DM's best solution among the candidate solutions on the Pareto-front, generated through NSGA-2. To this end, we employ a preference-based learning approach to learn an abstract ideal reference point of the DM on the multi-objective space, which reflects the compromises the DM makes against a set of conflicting objectives. The solution that is closest to this reference-point is then predicted as the DM's best solution. The pairwise comparisons of the candidate solutions provides the training information for our learning model. The experimental results on ZDT1 dataset shows that the proposed approach is not only intuitive, but also easy to apply, and robust to inconsistencies in the DM's preference statements.

关键词： Multi-objective optimization NSGA-2 preference-based learning Solution selection

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：