Abstract
The process of preparing potentially large and complex data sets for further analysis or manual examination is often called data wrangling. In classical warehousing environments, the steps in such a process have been carried
out using Extract-Transform-Load platforms, with significant manual involvement in specifying, configuring or tuning many of them. Cost-effective data wrangling processes need to ensure that data wrangling steps benefit from automation wherever possible. In this paper, we define a methodology to fully
automate an end-to-end data wrangling process incorporating data context, which associates portions of a target schema with potentially spurious extensional data of types that are commonly available. Instance-based evidence together with data profiling paves the way to inform automation in several steps
within the wrangling process, specifically, matching, mapping validation, value format transformation, and data repair. The approach is evaluated with real estate data showing substantial improvements in the results of automated wrangling.
out using Extract-Transform-Load platforms, with significant manual involvement in specifying, configuring or tuning many of them. Cost-effective data wrangling processes need to ensure that data wrangling steps benefit from automation wherever possible. In this paper, we define a methodology to fully
automate an end-to-end data wrangling process incorporating data context, which associates portions of a target schema with potentially spurious extensional data of types that are commonly available. Instance-based evidence together with data profiling paves the way to inform automation in several steps
within the wrangling process, specifically, matching, mapping validation, value format transformation, and data repair. The approach is evaluated with real estate data showing substantial improvements in the results of automated wrangling.
Original language | English |
---|---|
Title of host publication | 2017 IEEE International Conference on Big Data (Big Data) |
Publisher | IEEE |
Pages | 956-963 |
Number of pages | 8 |
ISBN (Electronic) | 978-1-5386-2715-0 |
ISBN (Print) | 978-1-5386-2716-7 |
DOIs | |
Publication status | Published - 15 Jan 2018 |
Event | 2017 IEEE International Conference on Big Data (Big Data) - Boston, United States Duration: 11 Dec 2017 → 14 Dec 2017 |
Conference
Conference | 2017 IEEE International Conference on Big Data (Big Data) |
---|---|
Country/Territory | United States |
City | Boston |
Period | 11/12/17 → 14/12/17 |
Keywords
- Data Context
- Data Integration
- Data Wrangling