检索结果-内蒙古大学图书馆

approximate dynamic programming in transportation and logistics: a unified framework

EURO JOURNAL ON TRANSPORTATION AND LOGISTICS 2012年第3期1卷 237-284页

作者： Powell, Warren B. Simao, Hugo P. Bouzaiene-Ayari, Belgacem Princeton Univ Dept Operat Res & Financial Engn Princeton NJ 08544 USA

Deterministic optimization has enjoyed a rich place in transportation and logistics, where it represents a mature field with established modeling and algorithmic strategies. By contrast, sequential stochastic optimization models (dynamic programs) have been plagued by the lack of a common modeling framework, and by algorithmic strategies that just do not seem to scale to real-world problems in transportation. This paper is designed as a tutorial of the modeling and algorithmic framework of approximate dynamic programming;however, our perspective on approximate dynamic programming is relatively new, and the approach is new to the transportation research community. We present a simple yet precise modeling framework that makes it possible to integrate most algorithmic strategies into four fundamental classes of policies, the design of which represents approximate solutions to these dynamic programs. The paper then uses problems in transportation and logistics to indicate settings in which each of the four classes of policies represents a natural solution strategy, highlighting the fact that the design of effective policies for these complex problems will remain an exciting area of research for many years. Along the way, we provide a link between dynamic programming, stochastic programming and stochastic search.

关键词： approximate dynamic programming dynamic programming Stochastic programming Stochastic search Logistics dynamic vehicle routing

来源：评论

学校读者我要写书评

暂无评论

Artificial intelligence as structural estimation: Deep Blue, Bonanza, and AlphaGo

引用

ECONOMETRICS JOURNAL 2020年第3期23卷 S1-S24页

作者： Igami, Mitsuru Yale Dept Econ 37 Hillhouse Ave New Haven CT 06511 USA

This article clarifies the connections between certain algorithms to develop artificial intelligence (AI) and the econometrics of dynamic structural models. with concrete examples of three 'game AIs'. Chess-playing Deep Blue is a calibrated value function, whereas shogiplaying Bonanza is an estimated value function via Rust's nested fixed-point (NFXP) method. AlphaGo's 'supervised-learning policy network' is a deep-neural-network implementation of the conditional-choice-probability (CCP) estimation reminiscent of Hotz and Miller's first step: the construction of its 'reinforcement-learning value network' is analogous to their conditional choice simulation (CCS). I then explain the similarities and differences between AI-related methods and structural estimation more generally, and suggest areas of potential cross-fertilization.

关键词： approximate dynamic programming artificial intelligence conditional choice probability deep neural network dynamic structural model inverse reinforcement learning optimal control reinforcement learning simulation estimator

来源：评论

学校读者我要写书评

暂无评论

Efficient algorithms of pathwise dynamic programming for decision optimization in mining operations

引用

ANNALS OF OPERATIONS RESEARCH 2020年第1-2期286卷 583-615页

作者： Hinz, Juri Tarnopolskaya, Tanya Yee, Jeremy Univ Technol Sydney Sydney NSW Australia CSIRO Canberra ACT Australia

Complexity and uncertainty associated with commodity resource valuation and extraction requires stochastic control methods suitable for high dimensional states. Recent progress in duality and trajectory-wise techniques has introduced a variety of fresh ideas to this field with surprising results. This paper presents a concept which implements this promising development and illustrates it on a selection of traditional commodity extraction problems. We describe efficient algorithms for obtaining approximate solutions along with a diagnostic technique, which provides a quantitative measure for solution performance in terms of the distance between the approximate and the optimal control policy. All quantitative tools are efficiently implemented and are publicly available within a user friendly package in the statistical language R, which can help practitioners in a broad range of decision optimization problems.

关键词： approximate dynamic programming Duality Markov decision process Natural resource extraction Optimal switching Real option

来源：评论

学校读者我要写书评

暂无评论

Learning solutions to hybrid control problems using Benders cuts 2

Learning solutions to hybrid control problems using Benders ...

引用

2nd Annual Conference on Learning for dynamics and Control (L4DC)

作者： Menta, Sandeep Warrington, Joseph Lygeros, John Morari, Manfred Swiss Fed Inst Technol Automat Control Lab Phys Str 3 CH-8092 Zurich Switzerland Univ Penn Elect & Syst Engn 220 S 33rd St Philadelphia PA 19104 USA

Hybrid control problems are complicated by the need to make a suitable sequence of discrete decisions related to future modes of operation of the system. Model predictive control (MPC) encodes a finite-horizon truncation of such problems as a mixed-integer program, and then imposes a cost and/or constraints on the terminal state intended to reflect all post-horizon behaviour. However, these are often ad hoc choices tuned by hand after empirically observing performance. We present a learning method that sidesteps this problem, in which the so-called N-step Q-function of the problem is approximated from below, based on experience evaluating the policy. The function takes a state and a sequence of N control decisions as arguments, and therefore extends the traditional notion of a Q-function from reinforcement learning. After learning it from a training process exploring the state-input space, we use it in place of the usual MPC objective. We take an example hybrid control task and show that it can be completed successfully with a shorter planning horizon than conventional hybrid MPC thanks to our proposed method. Furthermore, we report that Q-functions trained with long horizons can be truncated to a shorter horizon for online use, yielding simpler control laws with apparently little loss of performance.

关键词： Hybrid systems reinforcement learning approximate dynamic programming

来源：评论

学校读者我要写书评

暂无评论

A sequential decision and data analytics framework for maximizing value and reliability of CO2 storage monitoring

引用

JOURNAL OF NATURAL GAS SCIENCE AND ENGINEERING 2021年 96卷

作者： Tadjer, Amine Hong, Aojie Bratvold, Reidar B. Univ Stavanger Stavanger Norway

Carbon capture and sequestration (carbon capture and storage or CCS) represents a unique potential strategy that can minimize CO2 emissions in the atmosphere, and it creates a pathway toward a neutral carbon balance, which cannot be solely achieved by combining energy efficiency and other forms of low carbon energy. To contribute to the decision-making process and ensure that CCS is successful and safe, an adequate monitoring program must be implemented to prevent storage reservoir leakage and contamination of drinking water in groundwater aquifers. In this paper, we propose an approach to perform value of information (VOI) analyses to address sequential decision problems in reservoir management in the context of monitoring the geological storage of CO2 operations. These sequential decision problems are often solved and modeled by approximate dynamic programming (ADP), which is a powerful technique for handling complex large-scale problems and finding a near-optimal solution for intractable sequential decision-making. In this study, we tested machine learning techniques that fall within ADP to estimate the VOI and determine the optimal time to stop CO2 injections into the reservoir based on information from seismic surveys. This ADP approach accounts for both the effect of the information obtained before a decision and the effect of the information that might be obtained to support future decisions while significantly improving the timing, value of the decision, and uncertainty of the CO2 plume behavior, thereby significantly increasing economic performance. The Utsira saline aquifer west of Norway was used to exemplify ADP's ability to improve decision support regarding CO2 storage projects.

关键词： Value of information CO2 storage monitoring approximate dynamic programming Machine learning Time-lapse seismic monitoring

来源：评论

学校读者我要写书评

暂无评论

Lambda-Policy Iteration with Randomization for Contractive Models with Infinite Policies: Well-Posedness and Convergence 2

Lambda-Policy Iteration with Randomization for Contractive M...

引用

2nd Annual Conference on Learning for dynamics and Control (L4DC)

作者： Li, Yuchao Johansson, Karl H. Martensson, Jonas KTH Royal Inst Technol Div Decis & Control Syst Stockholm Sweden

dynamic programming models are used to analyze lambda-policy iteration with randomization algorithms. Particularly, contractive models with infinite policies are considered and it is shown that well-posedness of the lambda-operator plays a central role in the algorithm. The operator is known to be well-posed for problems with finite states, but our analysis shows that it is also well-defined for the contractive models with infinite states studied. Similarly, the algorithm we analyze is known to converge for problems with finite policies, but we identify the conditions required to guarantee convergence with probability one when the policy space is infinite regardless of the number of states. Guided by the analysis, we exemplify a data-driven approximated implementation of the algorithm for estimation of optimal costs of constrained linear and nonlinear control problems. Numerical results indicate potentials of this method in practice.

关键词： lambda-policy iteration approximate dynamic programming reinforcement learning

来源：评论

学校读者我要写书评

暂无评论

An approximate dynamic programming framework for modeling global climate policy under decision-dependent uncertainty

引用

COMPUTATIONAL MANAGEMENT SCIENCE 2012年第3期9卷 339-362页

作者： Webster, Mort Santen, Nidhi Parpas, Panos MIT Engn Syst Div 77 Massachusetts Ave Cambridge MA 02139 USA Imperial Coll London Dept Comp London SW7 2AZ England

Analyses of global climate policy as a sequential decision under uncertainty have been severely restricted by dimensionality and computational burdens. Therefore, they have limited the number of decision stages, discrete actions, or number and type of uncertainties considered. In particular, two common simplifications are the use of two-stage models to approximate a multi-stage problem and exogenous formulations for inherently endogenous or decision-dependent uncertainties (in which the shock at time t+1 depends on the decision made at time t). In this paper, we present a stochastic dynamic programming formulation of the dynamic Integrated Model of Climate and the Economy (DICE), and the application of approximate dynamic programming techniques to numerically solve for the optimal policy under uncertain and decision-dependent technological change in a multi-stage setting. We compare numerical results using two alternative value function approximation approaches, one parametric and one non-parametric. We showt hat increasing the variance of a symmetric mean-preserving uncertainty in abatement costs leads to higher optimal first-stage emission controls, but the effect is negligible when the uncertainty is exogenous. In contrast, the impact of decision-dependent cost uncertainty, a crude approximation of technology R&D, on optimal control is much larger, leading to higher control rates (lower emissions). Further, we demonstrate that themagnitude of this effect grows with the number of decision stages represented, suggesting that for decision-dependent phenomena, the conventional two-stage approximation will lead to an underestimate of the effect of uncertainty.

关键词： Climate policy analysis approximate dynamic programming Decision dependent uncertainty Stochastic dynamic programming Endogenous uncertainty

来源：评论

学校读者我要写书评

暂无评论

Data-based and secure switched cyber-physical systems

引用

SYSTEMS & CONTROL LETTERS 2021年 148卷

作者： Zhai, Lijing Vamvoudakis, Kyriakos G. Georgia Inst Technol Daniel Guggenheim Sch Aerosp Engn Atlanta GA 30332 USA

In this paper, we develop a completely model-free moving target defense framework for the detection and mitigation of sensor and/or actuator attacks in cyber-physical systems with dynamics that evolve in discrete-time. We incorporate an intrusion detection mechanism based on an approximate dynamic programming technique that learns the policies for optimal regulation and optimal tracking while simultaneously defending against actuator and sensor attacks in a model-free fashion. Switching rules are leveraged to force proactive and reactive defense mechanisms as well as, guarantee the stability of the equilibrium point. Finally, as a case study, we apply the proposed moving target defense framework to a DC-DC converter that is used in electric vehicles. (C) 2020 Elsevier B.V. All rights reserved.

关键词： Cyber-physical systems approximate dynamic programming Moving target defense Learning Security Switched systems

来源：评论

学校读者我要写书评

暂无评论

Adaptive network traffic control with an integrated model-based and data-driven approach and a decentralised solution method*

引用

TRANSPORTATION RESEARCH PART C-EMERGING TECHNOLOGIES 2021年 128卷

作者： Su, Z. C. Chow, Andy H. F. Zhong, R. X. Univ Hong Kong Dept Architecture & Civil Engn Hong Kong Peoples R China Sun Yat Sen Univ Sch Intelligent Syst Engn Guangzhou Peoples R China

This paper presents an adaptive traffic controller for stochastic road networks with an integrated model-based and data-driven solution framework. The model-based optimisation component operates based upon an underlying kinematic wave model driven by stochastic demand within a prediction horizon. The data-driven optimisation component operates based upon an approximate dynamic programming (ADP) formulation which approximates the state-control interactions over future stages with a parametric approximator. The approximator reduces the computational complexity of the adaptive control problem by parameterising the state and decision space. The parametric approximator is to be iteratively updated with online feeding of realisations of traffic states via a temporal difference (TD) learning process. Our results reveal that incorporation of the model-based component facilitates the training of the ADP-based state approximator, and hence improve the overall performance of the control system. We further develop a decentralised solution approach in which individual intersections are allowed to derive their own control policies in an asynchronous manner. The data-driven ADP approximator would serve as a central agent coordinating the control policies derived at individual intersections in the network. This is shown to be able to improve and stabilise the performance of the overall control system even under congested conditions. This is a significant progress in adaptive control system design with use of decentralised optimisation techniques. The present study contributes to the adaptive network traffic control with uncertainties through use of advanced modelling and optimisation methods.

关键词： Adaptive network traffic control Markov decision process approximate dynamic programming Acyclic signal timing Decentralised optimisation

来源：评论

学校读者我要写书评

暂无评论

Safe Policy Iteration: A Monotonically Improving approximate Policy Iteration Approach

引用

JOURNAL OF MACHINE LEARNING RESEARCH 2021年 22卷

作者： Metelli, Alberto Maria Pirotta, Matteo Calandriello, Daniele Restelli, Marcello Politecn Milan DEIB Milan Italy Facebook AI Res Paris France Ist Italiano Tecnol Genoa Italy

This paper presents a study of the policy improvement step that can be usefully exploited by approximate policy-iteration algorithms. When either the policy evaluation step or the policy improvement step returns an approximated result, the sequence of policies produced by policy iteration may not be monotonically increasing, and oscillations may occur. To address this issue, we consider safe policy improvements, i.e., at each iteration, we search for a policy that maximizes a lower bound to the policy improvement w.r.t. the current policy, until no improving policy can be found. We propose three safe policy-iteration schemas that differ in the way the next policy is chosen w.r.t. the estimated greedy policy. Besides being theoretically derived and discussed, the proposed algorithms are empirically evaluated and compared on some chain-walk domains, the prison domain, and on the Blackjack card game.

关键词： Reinforcement Learning approximate dynamic programming approximate Policy Iteration Policy Oscillation Policy Chattering Markov Decision Process

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：