Evaluating whole system reforms: A structured approach for selecting multiple outcomes

Whole-system reforms, including devolution and integration of health and social care services, have the potential to impact multiple dimensions of health system performance. Most evaluations focus on a single or narrow subsets of outcomes amenable to change. This approach may not: (i) capture the overall effect of the reform, (ii) identify the mechanisms through which system-wide changes may have occurred, (iii) prevent post-hoc selection of outcomes based on significant results; and (iv) facilitate comparisons across settings. We propose a structured approach for selecting multiple quantitative outcome measures, which we apply for evaluating health and social care devolution in Greater Manchester, England. The approach consists of five-steps: (i) defining outcome domains based on a framework, in our case the World Health Organisation’s Health System Performance Assessment Framework; (ii) reviewing performance metrics from national monitoring frameworks; (iii) excluding similar and condition specific outcomes; (iv) excluding outcomes with insufficient data; and (v) mapping implemented policies to identify a subset of targeted outcomes. We identified 99 outcomes, of which 57 were targeted. The proposed approach is detail and time-intensive, but useful for both researchers and policymakers to promote transparency in evaluations and facilitate the interpretation of findings and cross-settings comparisons.