Dataset Discovery in Data Lakes

Alex Bogatu, Alvaro Fernandes, Norman Paton, Nikolaos Konstantinou

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

112 Downloads (Pure)


Data analytics stands to benefit from the increasing availability of datasets that are held without their conceptual relationships being explicitly known. When collected, these datasets form a data lake from which, by processes like data wrangling, specific target datasets can be constructed that enable value– adding analytics. Given the potential vastness of such data lakes, the issue arises of how to pull out of the lake those datasets that might contribute to wrangling out a given target. We refer to this as the problem of dataset discovery in data lakes and this paper contributes an effective and efficient solution to it. Our approach uses features of the values in a dataset to construct hash–based indexes that map those features into a uniform distance space. This makes it possible to define similarity distances between features and to take those distances as measurements of relatedness w.r.t. a target table. Given the latter (and exemplar tuples), our approach returns the most related tables in the lake. We provide a detailed description of the approach and report on empirical results for two forms of relatedness (unionability and joinability) comparing them with prior work, where pertinent and showing significant improvements in all of precision, recall, target coverage, indexing and discovery times.
Original languageEnglish
Title of host publication36th IEEE International Conference on Data Engineering
Publication statusE-pub ahead of print - 27 May 2020
Event36th IEEE International Conference on Data Engineering - Dallas, United States
Duration: 20 Apr 202024 Apr 2020


Conference36th IEEE International Conference on Data Engineering
Country/TerritoryUnited States


  • data discovery
  • table search
  • data wrangling


Dive into the research topics of 'Dataset Discovery in Data Lakes'. Together they form a unique fingerprint.

Cite this