Skip to main content

Navigating policy complexity: a Delphi method approach

By Ben Szreter, Rachel Carter and Esme Yates

The Oracle of Delphi in Ancient Greece was considered a powerful source of prophecy and insight. Being able to predict the future would, of course, be the ultimate tool for policymaking. But while this is impossible, using a technique named after this ancient oracle – the Delphi method – could be an underutilised asset in policy development.

A Delphi approach can help identify people’s ideas about the future and calibrate those ideas against each other. The unique selling point of the Delphi method is that group decisions and analysis – without the biases that come with group hierarchies – can be better than an individual’s alone. This is as true in policy as in other areas and could be especially pertinent in complex policy landscapes where traditional approaches may fall short.

There have been examples of the Delphi method being used in policymaking from the 1970s, but its use has not been widespread. For example, it has previously been used in forecasting (on topics such as public policy issues, economic trends, health and education), and to help reach expert consensus in health-related settings, such as in the development of medical guidelines and protocols.

In the UK 2040 Options project, we have set out to draw on collective wisdom as we consider policy options that will get the UK to a better place by 2040. In some instances, we have used an adapted Delphi process to draw on a wide range of expert perspectives, which has given us a large funnel through which to filter the wide range of issues and ideas that we are exploring.

What is a Delphi exercise and how have we run them?

Our Delphi exercises included two rounds. In round one, we asked respondents about the issues and interventions that could impact the topic in question in the UK (such as economic growth) and to rank their responses by importance or potential impact. In round two, we fed these issues and interventions back to the respondents alongside their average importance or impact scores from round one, before asking the experts surveyed for their final judgement. Opinions of all experts were kept anonymous throughout.

The final output is a list of policy issues and interventions, ranked by the scores given by experts in round two.

This process can be seen in the below diagram.

 

How we’ve used the Delphi method to crowdsource expert views

We have run Delphi exercises on policy topics such as growth and productivity, education, health and net zero. In our adaptation of the Delphi method, we aimed to capture the multifaceted nature of policy issues surrounding various major policy problems in the UK.

Across these Delphis, we have been able to engage with a diverse group of more than 250 experts in total, identify well over 100 critical issues and propose over 900 potential interventions. Using this method has allowed us to get much richer information from experts and emerging thinkers, in a more useable way, than a traditional survey or roundtable alone.

What we gained from using Delphis in a policy context

The Delphi process is a good way to cast a wide net for different views and perspectives on a particular topic. It allowed us to quickly identify the most important issues that there is expert consensus around, as well as pinpoint a wide set of possible interventions.

This helped us to focus subsequent in-person discussions on policy priorities around areas that had been identified as being of the most concern and gave us an understanding of where there is convergence in opinion (the “no brainers”) and where there is divergence (a potential choice for policymakers to make). The suggested interventions also acted as a spark for discussions with experts around trade-offs, the feasibility of implementation, and where government attention should focus next.

All of this provided rich and invaluable insight into specific policy areas, which we were able to draw on and distil in our public-facing research. You can see examples of how we used the outputs of the Delphi process in our education, health and wealth and income inequality ‘Choices’ series.

Insights we gained into the Delphi model

Participants were able to comment on the user experience of the Delphi, and provide feedback on the method and approach. From this, we were able to identify a number of key considerations.

  • The wording of the questions needs to be succinct and well-considered. When working with people that have, at most, 10 minutes of focussed time, the research question has to be articulated quickly. Initially, the lack of specificity in some questions led to challenges in categorising responses for the second round of the Delphi, with some responses being very broad or vague (for example: income redistribution) and others being extremely specific (such as more effective limits on charging for school uniforms).
  • Pre-specification is really important. Respondents sometimes mentioned the overwhelming nature of the survey’s second stage, citing the large number of items as a barrier to providing meaningful responses. It is therefore important to understand from the outset of the Delphi process what data will be categorised and how the results from the first phase will transfer to the second stage. This helps avoid bias in the approach and also helps formulate the questions.
  • Be conscious of your sampling constraints. Given the nature of the task, there will sometimes be a low number of responses. And given the design of the survey, selection bias is likely to be present.

The feedback we received was instrumental in shaping our understanding of the Delphi process’s efficacy and limitations. It underscored the need for a delicate balance between comprehensive coverage of topics and the cognitive load on respondents.

We also learnt to categorise our issues, ensuring a clear distinction between causes, symptoms, and potential policy responses.

The Delphi method: a valuable policy tool

Despite its challenges, our experience with the Delphi method reaffirmed its value as a tool for policy analysis. It facilitated a structured yet flexible platform for gathering expert opinions, allowing us to quickly pinpoint areas of consensus and divergence. The method’s iterative nature, coupled with the anonymity of responses, also helped in reducing biases and promoting honest, uninhibited feedback.

Looking ahead, the Delphi method could hold significant promise as a versatile tool in the policymaker’s toolkit. Its application can be further refined to suit various policy domains, with modifications tailored to specific contexts and objectives.