# Explicit solution of relative entropy weighted control

@article{Bierkens2014ExplicitSO, title={Explicit solution of relative entropy weighted control}, author={Joris Bierkens and Hilbert J. Kappen}, journal={Syst. Control. Lett.}, year={2014}, volume={72}, pages={36-43} }

Abstract We consider the minimization over probability measures of the expected value of a random variable, regularized by relative entropy with respect to a given probability distribution. In the general setting we provide a complete characterization of the situations in which a finite optimal value exists and the situations in which a minimizing probability distribution exists. Specializing to the case where the underlying probability distribution is Wiener measure, we characterize finite… Expand

#### Figures and Topics from this paper

#### 23 Citations

Malliavin Calculus Method for Asymptotic Expansion of Dual Control Problems

- Mathematics, Computer Science
- SIAM J. Financial Math.
- 2013

We develop a technique based on Malliavin-Bismut calculus ideas, for asymptotic expansion of dual control problems arising in connection with exponential indifference valuation of claims, and with… Expand

Variational and optimal control representations of conditioned and driven processes

- Mathematics, Computer Science
- 2015

These interpretations of the driven process generalize and unify many previous results on maximum entropy approaches to nonequilibrium systems, spectral characterizations of positive operators, and control approaches to large deviation theory and lead to new methods for analytically or numerically approximating large deviation functions. Expand

Linear PDEs and eigenvalue problems corresponding to ergodic stochastic optimization problems on compact manifolds

- Mathematics
- 2016

Long term average or 'ergodic' optimal control problems on a compact manifold are considered. The problems exhibit a special structure which is typical of control problems related to large deviations… Expand

Stochastic Optimal Control as Non-equilibrium Statistical Mechanics: Calculus of Variations over Density and Current

- Mathematics, Physics
- ArXiv
- 2013

This work proposes derivation of the generalized gauge-invariant Hamilton-Jacobi-Bellman equation as a variation over density and current, suggest hydrodynamic interpretation and discusses examples, e.g., ergodic control of a particle-within-a-circle, illustrating non-equilibrium space-time complexity. Expand

Linear PDEs and eigenvalue problems corresponding to ergodic stochastic optimization problems on compact manifolds

- Mathematics, Physics
- 2013

We consider long term average or `ergodic' optimal control poblems with a special structure: Control is exerted in all directions and the control costs are proportional to the square of the norm of… Expand

Nonlinear stochastic receding horizon control: stability, robustness and Monte Carlo methods for control approximation

- Mathematics, Computer Science
- Int. J. Control
- 2018

An approximation method based on Monte Carlo simulation derived via the Feynman–Kac formula gives a stochastic interpretation for the solution of a Hamilton–Jacobi–Bellman equation associated with the true optimal controller. Expand

Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

- Computer Science, Mathematics
- ArXiv
- 2020

A principled framework based on divergences between path measures is developed, inspired by connections to forward-backward SDEs, and proposed and studied the novel log-variance divergence, showing favourable properties of corresponding Monte Carlo estimators. Expand

A Variational Formula for Risk-Sensitive Reward

- Mathematics, Computer Science
- SIAM J. Control. Optim.
- 2017

We derive a variational formula for the optimal growth rate of reward in the infinite horizon risk-sensitive control problem for discrete time Markov decision processes with compact metric state and… Expand

Adaptive Smoothing Path Integral Control

- Computer Science, Engineering
- J. Mach. Learn. Res.
- 2020

A model-free algorithm called ASPIC (Adaptive Smoothing of Path Integral Control) is proposed that applies an inf-convolution to the cost function to speedup convergence of policy optimization and shows analytically and empirically that intermediate levels of smoothing are optimal, which renders the new method superior to both PICE and direct cost-optimization. Expand

J an 2 01 5 A VARIATIONAL FORMULA FOR RISK-SENSITIVE REWARD

- 2018

We derive a variational formula for the optimal growth rate of reward in the infinite horizon risk-sensitive control problem for discrete time Markov decision processes with compact metric state and… Expand

#### References

SHOWING 1-10 OF 43 REFERENCES

Malliavin Calculus Method for Asymptotic Expansion of Dual Control Problems

- Mathematics, Computer Science
- SIAM J. Financial Math.
- 2013

We develop a technique based on Malliavin-Bismut calculus ideas, for asymptotic expansion of dual control problems arising in connection with exponential indifference valuation of claims, and with… Expand

A variational representation for certain functionals of Brownian motion

- Mathematics
- 1998

In this paper we show that the variational representation - log Ee -f(W) = inf E{1/2∫ 0 1 ∥v s ∥ 2 ds + f(w + ∫ 0 v s ds)} holds, where W is a standard d-dimensional Brownian motion, f is any bounded… Expand

Meshless discretization of LQ-type stochastic control problems

- Mathematics
- 2013

We propose a novel Galerkin discretization scheme for stochastic optimal control problems on an indefinite time horizon. The control problems are linear-quadratic in the controls, but possibly… Expand

Malliavin calculus for Levy processes with applications to finance

- Economics
- 2004

Extending Gaussian Malliavin derivatives, to a finite moments Levy process framework, using chaos expansions has proven to be a successful approach. In this work the theory is extended by the… Expand

A weak convergence approach to the theory of large deviations

- Mathematics
- 1997

Formulation of Large Deviation Theory in Terms of the Laplace Principle. First Example: Sanov's Theorem. Second Example: Mogulskii's Theorem. Representation Formulas for Other Stochastic Processes.… Expand

A Unifying Framework for Linearly Solvable Control

- Computer Science, Mathematics
- UAI
- 2011

This work generalizes the recently developed risk-sensitive path-integral control formalism and demonstrates the usefulness of the framework on control problems with noise where different values of lead to qualitatively different control behaviors. Expand

Linearly-solvable Markov decision problems

- Computer Science, Mathematics
- NIPS
- 2006

A class of MPDs which greatly simplify Reinforcement Learning, which have discrete state spaces and continuous control spaces and enable efficient approximations to traditional MDPs. Expand

Malliavin Calculus for Lévy Processes with Applications to Finance

- Mathematics
- 2008

The Continuous Case: Brownian Motion.- The Wiener-Ito Chaos Expansion.- The Skorohod Integral.- Malliavin Derivative via Chaos Expansion.- Integral Representations and the Clark-Ocone formula.- White… Expand

Controlled Markov processes and viscosity solutions

- Mathematics
- 1992

This book is intended as an introduction to optimal stochastic control for continuous time Markov processes and to the theory of viscosity solutions. The authors approach stochastic control problems… Expand

Logarithmic Transformations and Stochastic Control

- Mathematics
- 1982

Abstract : We are concerned with a class of problems described in a somewhat imprecise way as follows. Consider a linear operator of the form L + V(x), where L is the generator of a Markov process x… Expand