Dave Pankhurst - The use of Gaussian process emulators for improving the performance of benefit forecasting models

  • Presenting author: Dave Pankhurst (Department for Work and Pensions, UK)

  • Authors: Howard Redway, Steve Webster & Ian Vernon

  • Session: C04D - Uncertainty - Wednesday 16:00-17:00 - Erika-Weinzierl Hall

  • Slides: PDF

Gaussian Emulators (GEs) have been used to emulate parts of large complex and slow running models in science, including meteorology and cosmology. A GE is a statistical model of a model, usually with much reduced input parameters and outputs. Model emulators are designed to run much faster than the main model and can quickly predict the outputs for a large number of different parameter values and combinations; this opens the door to further applications to enhance model development. The UK Department for Work and Pensions (DWP) is responsible for over £230 billion of benefit and pension expenditure. It uses a range of dynamic microsimulation models to forecast expenditure and assess proposed policy changes. It is in this context that DWP is developing the application of GE methodology to enhance its microsimulation models. The work is being undertaken by DWP’s Gaussian Emulation Project in collaboration with experts in the field from Durham University who have provided advice and guidance on emulator methodology. This innovative project is demonstrating that GE and related applications have real potential to support improvements in the performance of complex microsimulation models used in government. This presentation will outline the Project’s progress and key results in using model emulators within DWP, including:

  1. How we successfully demonstrated the potential for GE to make a difference and are now applying these techniques to operational models.

  2. What do we use model emulators for? With examples including: (a) parameter sensitivity analysis (identifying the most important parameters and effects of interactions between parameters); and (b) ‘target fitting’ (adjusting parameters of the main model to better fit external targets or the most recent outturn, e.g. if circumstances are changing rapidly).

  3. Demonstrating the advantages of using emulators that have two elements: linear model + Gaussian process, using methodology as argued in Vernon, Ian., Goldstein, Michael. & Bower, Richard G. (2010). “Galaxy Formation: a Bayesian Uncertainty Analysis.” Bayesian Analysis 05(04): 619-670.

  4. Evaluating the impact of stochasticity and the implications for using model emulators. Investigating and developing methods for controlling stochasticity, including the ‘MultiStream’ method (see associated abstract ‘Using “MultiStream” to control uncertainty to model changes’ by Howard Redway and Steve Webster).