Effect of Continuous Integration on Build Health in Undergraduate Team Projects

We present the results of an analysis of the changing patterns
of build health quality across 3 cohorts of undergraduate students, in a
compulsory software engineering course unit. In the course unit, student
teams were asked to make changes to a large open source software system,
and to maintain clean release builds as they did so. Release build health
(in terms of compiling code and passing unit tests) was explicitly included
in the marking scheme for the coursework.
We set up a continuous integration server to keep track of student build
health. Initially, this was used only by TAs in marking student work,
but for later cohorts we provided access to continuous integration results
from the early stages of each exercise. This has provided us with data
on the changing patterns of student build health, with differing access to
the CI server, giving an insight into how students learn these skills and
the effects of allowing them access to CI results.
We found evidence of a clear improvement in ability to manage build
health when CI facilities are made available, but that some student teams
were not making use of the facilities to much effect. The improvement
effect was strongest on the build health of release builds, corresponding
to the area of greatest marks in the marking scheme. The CI results also
proved to be very valuable for academic staff, in making the problems
with student builds visible.