different energy sources at future dates (will fusion be available, will large-scale storage
devices be available, etc.), and many others. In all these cases we lack a widely-
agreed-upon framework for making decisions.
Stochastic Optimization, Statistics and Machine Learning
There are numerous problems in the design and control of energy devices,
networks and markets that need to be modeled as sequential decision problems in the
presence of different forms of uncertainty. Decisions include storing/withdrawing energy
from a battery, determining which energy generators to use each hour, how to price
recharging stations for electric vehicles, optimal maintenance of grid components,
optimal load curtailment to meet grid capacity constraints, design of energy investment
portfolios and robust design of the power grid. Other stochastic optimization problems
arise in laboratory environments: how to sequence the testing of new compounds,
optimal design of experiments, and optimal sampling of materials and processes to
obtain the best performance from new materials for converting biomass.
Stochastic optimization is an intrinsically difficult problem, as it involves the
sequential choice of decisions (controls), followed by observations of new information,
followed by more decisions. Unless the problem has special structure, the research
community has focused on three broad strategies: 1) lookahead policies, which include
tree-search and stochastic programming (Birge and Louveaux, 1997), 2) policy function
approximations, which involves searching within a well defined class of functions, and 3)
policies based on approximating the value function in Bellman’s equation.
We do not have general purpose algorithms for finding optimal policies, and we
often struggle even to find good policies. The field of approximate dynamic
programming (known as reinforcement learning in computer science) blends simulation,
deterministic math programming and machine learning (to approximate the value
function), producing some successes (Bertsekas and Tsitsiklis, 1996, Powell, 2007,
Sutton and Barto,1998). There is active research in the design of all three types of
policies listed above. The complexity of lookahead policies grows exponentially with the
number of time periods. Policy function and value function introduce the difficult
challenge of specifying and fitting functions, introducing a range of challenges to the
statistics and machine learning communities (Hastie, et al 2009).
This discussion ignores important modeling issues in the handling of uncertainty. For
example, electricity prices are easy to quantify, but are not described by standard
Gaussian distributions and have been found to be heavy-tailed (infinite variance), which
means that you cannot compute an expectation. There are a number of instances
where we would like to introduce risk as an explicit constraint, such as the risk that we
will not meet a renewable target, or the risk that we will overuse a backup diesel
generator. There are also problems where the uncertainty is hard to model, such as the
likelihood that Congress will pass a tax on carbon or that there will be a breakthrough in
batteries.
For the near term, there is active research simply to solve narrow problem
classes. A longer term goal is to develop robust, general purpose tools that solve
broader problem classes. For example, parametric approximations of value or policy
functions are the easiest to estimate, but introduce an undesirable manual step in the
design of these functions. Nonparametric techniques offer considerable generality, but
these are harder to use and still struggle with functions with even a modest number of