SACCADE

Semantic-Aware Compactor for Curating Data Lakes

Our proposal SACCADE received funding from the Swedish Research Council (VR) call for research projects within natural and engineering sciences.

AI/ML-enabled systems need large amounts of high quality data for training and evaluation. At industrial scale, such datasets are already referred to as data lakes to denote their size. Growing them by just adding more and more data drops is rather simple. However, determining the added value of the next drop to be inserted to a data lake has become increasingly complicated.

In SACCADE, we aim at investigating, developing, and systematically evaluating an automated approach to estimate the added value of new data to be inserted to a large-scale data lake in a computationally efficient way. We will exploit space-filling curves (SFC) to map multi-dimensional data as well as unstructured large multimodal data to their corresponding, single dimensional representations. For the latter, we will embody a deep learning (DL) approach to extract semantic-aware key features as input to SFCs.

We will use patterns emerging on SFCs to discover already present data samples and also to identify characteristics of still missing data samples in a data lake. Thereby, we expect that it will be possible to determine the level of data diversity and quality in existing data lakes to tackle important questions such as “How good are our datasets and when do we have enough data?”