检索结果-内蒙古大学图书馆

HILC: Domain-Independent PbD System Via Computer Vision and Follow-Up Questions

ACM TRANSACTIONS ON INTERACTIVE INTELLIGENT SYSTEMS 2019年第2-3期9卷 16-16页

作者： Intharah, Thanapong Turmukhambetov, Daniyar Brostow, Gabriel J. UCL Gower St London WC1E 6BT England Khon Kaen Univ 123 Mittraphap Rd Khon Kaen 40002 Thailand

Creating automation scripts for tasks involving Graphical User Interface (GUI) interactions is hard. It is challenging because not all software applications allow access to a program's internal state, nor do they all have accessibility APIs. Although much of the internal state is exposed to the user through the GUI, it is hard to programmatically operate the GUI's widgets. To that end, we developed a system prototype that learns by demonstration, called HILC (Help, It Looks Confusing). Users, both programmers and non-programmers, train HILC to synthesize a task script by demonstrating the task. A demonstration produces the needed screenshots and their corresponding mouse-keyboard signals. After the demonstration, the user answers follow-up questions. We propose a user-in-the-loop framework that learns to generate scripts of actions performed on visible elements of graphical applications. Although pure programming by demonstration is still unrealistic due to a computer's limited understanding of user intentions, we use quantitative and qualitative experiments to show that non-programming users are willing and effective at answering follow-up queries posed by our system, to help with confusing parts of the demonstrations. Our models of events and appearances are surprisingly simple but are combined effectively to cope with varying amounts of supervision. The best available baseline, Sikuli Slides, struggled to assist users in the majority of the tests in our user study experiments. The prototype with our proposed approach successfully helped users accomplish simple linear tasks, complicated tasks (monitoring, looping, and mixed), and tasks that span across multiple applications. Even when both systems could ultimately perform a task, ours was trained and refined by the user in less time.

关键词： programming by demonstration GUI automation action segmentation and recognition visual-based programming by demonstration

来源：评论

学校读者我要写书评

暂无评论

Instrumented Tool based Robot programming - Parameterization of Screwing Process Macros

引用

Procedia Manufacturing 2019年 38卷 415-422页

作者： Markus Ikeda Markus Ganglbauer Prateek Ashok Srinivas Maddukuri Michael Hofmann Andreas Pichler PROFACTOR GmbH. Im Stadtgut A2 AT-4407 Steyr-Gleink Austria

programming by demonstration (PbD), is a powerful mechanism for reducing the complexity of search space for learning. Current approaches for skill representation can be broadly divided between two trends. Low-level representations of skills, taking into account the form of a non-linear mapping between sensory and motor information, which is referred to as “trajectories encoding”. On the other hand high-level representations of skills that decompose them in a sequence of action-perception units which is referred to as “symbolic encoding” [1]. While for symbolic encoding visual perception of states and actions is one preferred approach trajectories usually are acquired from direct demonstration by hand guidance of robot arms carrying robotic tools by skilled workers. This approach requires both the process knowledge of the experienced worker and the possibility of intuitive hand guidance of the robotic tool which is negatively affected by the robots limited working space, friction or the reachability of the robot. Therefore and because it is also highly unintuitive to parameterize macro based robot programs with numerical values considering process parameters (like required forces or position parameters) we propose to let the worker use ordinary power tools during demonstration which are instrumented with additional sensors in order to be able to measure relevant parameters, so called “instrumented tools”. The paper presents the concept for a robotic workstation to be programmed by demonstration with an instrumented tool, the components of the instrumented tool as well as relevant calibration steps and results from application of the technology to an automotive preassembly process..

关键词： Collaborative Robotics in Smart Manufacturing programming by demonstration Zero programming Instrumented Tool Screwing Operations

来源：评论

学校读者我要写书评

暂无评论

Rousillon: Scraping Distributed Hierarchical Web Data 18

Rousillon: Scraping Distributed Hierarchical Web Data

引用

31st Annual ACM Symposium on User Interface Software and Technology (UIST)

作者： Chasins, Sarah E. Mueller, Maria Bodik, Rastislav UC Berkeley CA 94720 USA Univ Washington Seattle WA 98195 USA

ISBN: (纸本)9781450359481

programming by demonstration (PBD) promises to enable data scientists to collect web data. However, in formative interviews with social scientists, we learned that current PBD tools are insufficient for many real-world web scraping tasks. The missing piece is the capability to collect hierarchically-structured data from across many different webpages. We present Rousillon, a programming system for writing complex web automation scripts by demonstration. Users demonstrate how to collect the first row of a 'universal table' view of a hierarchical dataset to teach Rousillon how to collect all rows. To offer this new demonstration model, we developed novel relation selection and generalization algorithms. In a within-subject user study on 15 computer scientists, users can write hierarchical web scrapers 8 times more quickly with Rousillon than with traditional programming.

关键词： end-user programming programming by demonstration generalization data science web scraping program synthesis

来源：评论

学校读者我要写书评

暂无评论

RecurBot: Learn to Auto-complete GUI Tasks From Human demonstrations

RecurBot: Learn to Auto-complete GUI Tasks From Human Demons...

引用

CHI Conference on Human Factors in Computing Systems (CHI)

作者： Intharah, Thanapong Firman, Michael Brostow, Gabriel J. UCL Gower St London England

ISBN: (纸本)9781450356213

On the surface, task-completion should be easy in graphical user interface (GUI) settings. In practice however, different actions look alike and applications run in operating-system silos. Our aim within GUI action recognition and prediction is to help the user, at least in completing the tedious tasks that are largely repetitive. We propose a method that learns from a few user-performed demonstrations, and then predicts and finally performs the remaining actions in the task. For example, a user can send customized SMS messages to the first three contacts in a school's spreadsheet of parents;then our system loops the process, iterating through the remaining parents. First, our analysis system segments the demonstration into discrete loops, where each iteration usually included both intentional and accidental variations. Our technical innovation approach is a solution to the standing motif-finding optimization problem, but we also find visual patterns in those intentional variations. The second challenge is to predict subsequent GUI actions, extrapolating the patterns to allow our system to predict and perform the rest of a task. We validate our approach on a new database of GUI tasks, and show that our system usually (a) gleans what it needs from short user demonstrations, and (b) auto-completes tasks in diverse GUI situations.

关键词： programming by demonstration Recurrent Action Recognition GUI programming

来源：评论

学校读者我要写书评

暂无评论

Intuitive, iterative and assisted virtual guides programming for human-robot comanipulation

Intuitive, iterative and assisted virtual guides programming...

引用

作者： Susana Sanchez Restrepo Universite Toulouse 3 Paul Sabatier (UT3 Paul Sabatier)

学位级别：博士

For a very long time, automation was driven by the use of tra- ditional industrial robots placed in cages, programmed to repeat more or less complex tasks at their highest speed and with maximum accuracy. This robot-oriented solution is heavily dependent on hard automation which requires pre-specified fixtures and time consuming programming, hindering robots from becoming flexible and versatile tools. These robots have evolved towards a new generation of small, inexpensive, inherently safe and flexible systems that work hand in hand with humans. In these new collaborative workspaces the human can be included in the loop as an active agent. As a teacher and as a co-worker he can influence the decision-making process of the robot. In this context, virtual guides are an important tool used to assist the human worker by reducing physical effort and cognitive overload during tasks accom- plishment. However, the construction of virtual guides often requires expert knowledge and modeling of the task. These limitations restrict the usefulness of virtual guides to scenarios with unchanging constraints. To overcome these challenges and enhance the flexibility of virtual guides programming, this thesis presents a novel approach that allows the worker to create virtual guides by demonstration through an iterative method based on kinesthetic teaching and displacement splines. Thanks to this approach, the worker is able to iteratively modify the guides while being assisted by them, making the process more intuitive and natural while reducing its painfulness. Our approach allows local refinement of virtual guiding trajectories through physical interaction with the robots. We can modify a specific cartesian keypoint of the guide or re-demonstrate a portion. We also extended our approach to 6D virtual guides, where displacement splines are defined via Akima interpolation (for translation) and quadratic interpolation of quaternions (for orientation). The worker can initially define a

关键词： Virtual guides programming by demonstration collaborative robotics comanipulation

来源：评论

学校读者我要写书评

暂无评论

A Reconfigurable Pick-Place System Under Robot Operating System 11th

A Reconfigurable Pick-Place System Under Robot Operating Sys...

引用

11th International Conference on Intelligent Robotics and Applications (ICIRA)

作者： Ding, Cheng Wu, Jianhua Xiong, Zhenhua Liu, Chao Shanghai Jiao Tong Univ 800 Dongchuan Rd Shanghai 200240 Peoples R China

ISBN: (纸本)9783319975894;9783319975887

Pick-place systems are normally programmed manually to finish a task. If work changes, the system has to be reprogrammed which is cumbersome and time-consuming. This paper takes advantage of programming by demonstration (PbD) concept to simplify the programming process. Specifically, we designed a pick-place system under robot operating system (ROS) which could track the teacher's demonstration via object detection and estimate the 6D pose of the object through iterative closest point algorithm. Then the robot could imitate the task through a PbD technique, symbolic reasoning. The experiment shows that the system is reconfigurable and a new pick-place task could be programmed in thirty seconds.

关键词： programming by demonstration Robot operating system Pick-place

来源：评论

学校读者我要写书评

暂无评论

An Intuitive Robot Learning from Human demonstration 11th

An Intuitive Robot Learning from Human Demonstration

引用

11th International Conference on Intelligent Robotics and Applications (ICIRA)

作者： Ogenyi, Uchenna Emeoha Zhang, Gongyue Yang, Chenguang Ju, Zhaojie Liu, Honghai Univ Portsmouth Sch Comp Portsmouth Hants England Swansea Univ Coll Engn Swansea W Glam Wales

ISBN: (纸本)9783319975863;9783319975856

This paper presents a new way to teach a robot certain motions remotely from human demonstrator. The human and robot interface is built using a Kinect sensor which is connected directly to a remote computer that runs on processing software. The Cartesian coordinates are extracted, converted into joint angles and sent to the workstation for the control of the Sawyer robot. Kinesthetic teaching was used to correct the reproduced demonstrations while only valid resolved joint angles are recorded to ensure consistence in the sent data. The recorded dataset is encoded using GMM while GMR was employed to extract and reproduce generalised trajectory with respect to the associated time-step. To evaluate the proposed approach, an experiment for a robot to follow a human arm motion was performed. This proposed approach could help non-expert users to teach a robot how to perform assembling task in more human like ways.

关键词： programming by demonstration Learning from demonstration Learning by imitation

来源：评论

学校读者我要写书评

暂无评论

Automatic extraction of constraints in manipulation tasks for autonomy and interaction

Automatic extraction of constraints in manipulation tasks fo...

引用

作者： Ana-Lucia URECHE ECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE

学位级别：博士

Tasks routinely executed by humans involve sequences of actions performed with high dexterity and coordination. Fully specifying these actions such that a robot could replicate the task is often difficult. Furthermore the un- certainties introduced by the use of different tools or changing configurations demand the specification to be generic, while enhancing the important task as- pects, i. e. the constraints. Therefore the first challenge of this thesis is inferring these constraints from repeated demonstrations. In addition humans explaining a task to another person rely on the persons ability to apprehend missing or im- plicit information. Therefore observations contain user-specific cues, alongside knowledge on performing the task. Thus our second challenge is correlating the task constraints with the user behavior for improving the robots performance. We address these challenges using a programming by demonstration framework. In the first part of the thesis we describe an approach for decomposing demonstrations into actions and extracting task-space constraints as continuous features that apply throughout each action. The constraints consist of: (1) the reference frame for performing manipulation, (2) the variables of interest relative to this frame, allowing a decomposition in force and position control, and (3) a stiffness gain modulating the contribution of force and position. We then extend this approach to asymmetrical bimanual tasks by extracting features that enable arm coordination: the master–slave role that enables precedence, and the motion–motion or force–motion coordination that facilitates the physical interaction through an object. The set of constraints and the time–independent encodings of each action form a task prototype, used to execute the task. In the second part of the thesis we focus on discovering additional features implicit in the demonstrations with respect to two aspects of the teaching in- teractions: (1) characterizing the user p

关键词： programming by demonstration constraints extraction bimanual tasks coordination constraints learning and adaptive systems robot facial displays interaction dynamics interaction metrics human factors

来源：评论

学校读者我要写书评

暂无评论

Automatic Diagnosis of Students' Misconceptions in K-8 Mathematics

Automatic Diagnosis of Students' Misconceptions in K-8 Mathe...

引用

CHI Conference on Human Factors in Computing Systems (CHI)

作者： Feldman, Molly Q. Cho, Ji Yong Ong, Monica Gulwani, Sumit Popovic, Zoran Andersen, Erik Cornell Univ Dept Comp Sci Ithaca NY 14853 USA Univ Penn Dept Comp & Informat Sci Philadelphia PA 19104 USA Microsoft Res Redmond Redmond WA USA Univ Washington Paul G Allen Sch Comp Sci & Engn Seattle WA 98195 USA Cornell Univ Ithaca NY 14853 USA

ISBN: (纸本)9781450356206

K-8 mathematics students must learn many procedures, such as addition and subtraction. Students frequently learn "buggy" variations of these procedures, which we ideally could identify automatically. This is challenging because there are many possible variations that reflect deep compositions of procedural thought. Existing approaches for K-8 math use manually specified variations which do not scale to new math algorithms or previously unseen misconceptions. Our system examines students' answers and infers how they incorrectly combine basic skills into complex procedures. We evaluate this approach on data from approximately 300 students. Our system replicates 86% of the answers that contain clear systematic mistakes (13%). Investigating further, we found 77% at least partially replicate a known misconception, with 53% matching exactly. We also present data from 29 participants showing that our system can demonstrate inferred incorrect procedures to an educator as successfully as a human expert.

关键词： programming by demonstration elementary education

来源：评论

学校读者我要写书评

暂无评论

Reactile: programming Swarm User Interfaces through Direct Physical Manipulation

Reactile: Programming Swarm User Interfaces through Direct P...

引用

CHI Conference on Human Factors in Computing Systems (CHI)

作者： Suzuki, Ryo Kato, Jun Gross, Mark D. Yeh, Tom Univ Colorado Boulder Boulder CO 80309 USA Natl Inst Adv Ind Sci & Technol Tsukuba Ibaraki Japan

ISBN: (纸本)9781450356206

We explore a new approach to programming swarm user interfaces (Swarm UI) by leveraging direct physical manipulation. Existing Swarm UI applications are written using a robot programming framework: users work on a computer screen and think in terms of low-level controls. In contrast, our approach allows programmers to work in physical space by directly manipulating objects and think in terms of high-level interface design. Inspired by current UI programming practices, we introduce a four-step workflow create elements, abstract attributes, specify behaviors, and propagate changes for Swarm UI programming We propose a set of direct physical manipulation techniques to support each step in this workflow. To demonstrate these concepts, we developed Reactile, a Swarm UI programming environment that actuates a swarm of small magnets and displays spatial information of program states using a DLP projector. Two user studies-an in-class survey with 148 students and a lab interview with eight participants confirm that our approach is intuitive and understandable for programming Swarm UIs.

关键词： swarm user interfaces tangible programming programming by demonstration direct manipulation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：