VADA: An Architecture for End User Informed Data Preparation

Background: Data scientists spend considerable amounts of time preparing data
for analysis. Data preparation is labour intensive because the data scientist
typically takes fine grained control over each aspect of each step in the process,
motivating the development of techniques that seek to reduce this burden.

Results: This paper presents an architecture in which the data scientist need
only describe the intended outcome of the data preparation process, leaving the
software to determine how best to bring about the outcome. Key wrangling
decisions on matching, mapping generation, mapping selection, format
transformation and data repair are taken by the system, and the user need only
provide: (i) the schema of the data target; (ii) partial representative instance data
aligned with the target; (iii) criteria to be prioritised when populating the target;
and (iv) feedback on candidate results. To support this, the proposed architecture dynamically orchestrates a collection of loosely coupled wrangling components, in which the orchestration is declaratively specied and includes self-tuning of component parameters.

Conclusion: This paper describes a data preparation architecture that has been
designed to reduce the cost of data preparation through the provision of a central role for automation. An empirical evaluation with deep web and open government data investigates the quality and suitability of the wrangling result, the cost-effectiveness of the approach, the impact of self-tuning, and scalability with respect to the numbers of sources.