Multilevel Modelling in practice: measuring value added for school accountability in Chile

In the last few decades, school value-added studies have largely demonstrated the effects of socio-economic and demographic characteristics of the schools and the pupils on academic performance. Traditionally, contextualised value-added (CVA) models have been employed in educational research to assess variation in learning outcomes (usually standardised tests of Mathematics and Language) arising from schools and pupils, by implementing multilevel statistical models (2-level models). These models can also be used (and have been used) for informing school accountability to various stakeholders, including policy-makers, head teachers, parents and carers, etc.Despite an overwhelming amount of evidence in favour of the use of such models, government practice in Chile has been restricted to the use of raw school averages in standardised tests (SIMCE tests) as indicators of effectiveness. Nevertheless, a new accountability system that takes into account some of the principles of value-added research is being developed and is soon to be piloted in 2015 by the newly created Chilean Agency for the Quality of Education. The main issue with this new system is that it has been set out to implement multiple linear regression (MLR) models to adjust school averages, which is undoubtedly a major setback. This issue is critical given the high stakes; schools performing insufficiently, as judged by this system, could face closure.Using data from the Chilean National Pupil Database (SIMCE 2004-2006), this study demonstrates that the traditional value-added models (2-level CVA models), let alone MLR models, fall short in addressing the complex phenomenon of academic performance, especially in the context of a developing country. This is because they rather unrealistically assume, amongst other aspects, that a) only the last school attended affects pupils' performance; b) there are no differences between classes within schools; c) school effectiveness does not vary across areas; and d) there are no relationships between different subjects. Ignoring these untenable assumptions results in a large overestimation of the school effects. The novelty of the CVA models in this study is that they extend and improve the traditional 2-level school value-added models insofar as they explicitly assess the variation between pupils, classrooms, primary schools, secondary schools and local authorities, as well as the correlation between Mathematics and Language at all levels.After fitting the extended and more complex CVA models, school accountability measures are derived from them and compared to measures derived from simpler models. A number of key differences are found, leading to the conclusion that further adjustments to the traditional CVA models are not negligible, especially in a highly unequal country such as Chile. The choice of methods for school accountability is therefore a key player in this high stakes game and should not be taken as a trivial matter. A fair policy of school accountability should take a thorough account of the complexity of the school performance phenomenon.