Investigating Locally Commissioned Evaluations of the NHS Vanguard Programme

Background
In England, the 2012 Health and Social Care Act made innovation in the provision of health services a statutory duty with further impetus for major system change set out in the Five Year Forward View in 2014. With innovation in service delivery increasingly viewed as crucial to the long term sustainability of health systems, NHS England launched the Vanguard programme in 2015. Fifty Vanguard sites were to act as pilots for multicomponent innovations in service delivery, supported by a £200 million transformation fund from NHS England.

As part of the national evaluation of the New Models of Care Vanguard Programme in England led by researchers at the University of Manchester, we conducted an evidence synthesis to assess the nature and quality of locally commissioned evaluations relating to three vanguard types, namely enhanced health in care homes, the primary and acute care systems (PACS) and the Multispecialty community providers (MCPs).

Methods
The FutureNHS collaboration platform (Kahootz) is a secure, online hub used by the Vanguard and other integrated care initiatives. The repository was designated as the means by which vanguards could store, share and access key documents in one central hub. Two researchers accessed Kahootz every month from June 2017 to September 2018 to search for documents relevant to the synthesis. In addition, to Kahootz, we also searched for eligible evaluation reports on vanguard and named evaluator websites. Any report or slide set from a locally commissioned evaluation of a vanguard was eligible for inclusion. All identified documents were downloaded logged and details of the evaluators, questions, methodological approaches and limitations in design and/or reporting were extracted. As the included evaluations were largely mixed methods with variation in the nature and type of quantitative, qualitative and cost components, we performed a narrative synthesis of the evidence. Consistent with an integrative approach to synthesising evidence the narrative synthesis aimed to present a descriptive summary of the nature, type and general quality of evaluations within, and then to generate across vanguard types, a number of themes relevant to the aims of this review. The synthesis is supplemented with semi structured interviews with local evaluation leads from a purposive sample of Vanguard sites.

Results
We identified a total of 108 separate reports relating to the locally commissioned evaluations. Most were both resource and time limited and as a consequence, often attempted but did not fully address the wide range of questions posed in individual tender specifications. Most employed a mix of methods and largely focused on describing the implementation context and capturing stakeholder reflections on and experience of the development and implementation of individual Vanguards. Some evaluators struggled to access routine data on health service utilisation and other performance metrics which has hampered the conduct of analyses. Although a majority stated intentions to capture patient experience and conduct ‘economic’ or ‘cost’ related analysis, a combination of resource, data, time constraints mean that these components often lack depth, are often not fully realised or not conducted at all. We have identified a number of common methodological challenges and limitations across the local evaluations

Conclusions
A significant investment was made in independent local evaluations of the vanguard programme by NHS England. This synthesis represents the first attempt to systematically assess the nature and quality of the evaluations commissioned and to capture methodological learning to inform future endeavours of this type. As the synthesis summarises a significant grey literature not all of which is publicly available, this review represents the only comprehensive mapping of this knowledge base and may serve as a key resource for researchers and policy makers investigating large scale system change, both within the NHS and internationally.