From Big Data to Fast Data: Efficient Stream Data Management

Du Big Data au Fast Data: Gestion efficace des données de flux

Alexandru Costan (1, 2)

1 INSA Rennes - Institut National des Sciences Appliquées - Rennes
2 KerData - Scalable Storage for Clouds and Beyond

This manuscript provides a synthetic overview of my research journey since my PhD defense. The document does not claim to present my work in its entirety, but focuses on the contributions to data management in support of stream processing. These results address all stages of the stream processing pipeline: data collection and in-transit processing at the edge, transfer towards the cloud processing sites, ingestion and persistent storage. I start by presenting the general context of stream data management in light of the recent transition from Big to Fast Data. After highlighting the challenges at the data level associated with batch and real-time analytics, I introduce a subjective overview of my proposals to address them. They bring solutions to the problems of in-transit stream storage and processing, fast data transfers, distributed metadata management, dynamic ingestion and transactional storage. The integration of these solutions into functional prototypes and the results of the large-scale experimental evaluations on clusters, clouds and supercomputers demonstrate their effectiveness for several real-life applications ranging from neuro-science to LHC nuclear physics. Finally, these contributions are put into the perspective of the High Performance Computing - Big Data convergence.

Mots clés: data analytics; Big Data; metadata management; data management; in-transit processing; workflow management; HPC; stream processing; data transfers; storage

Mots clés: gestion de données; transactions; analyse de données; traitement de flux; gestion de métadonnées; transfert de données; données massives

Domaines: Calcul parallèle, distribué et partagé [cs.DC]; Base de données [cs.DB]; Algorithme et structure de données [cs.DS]; Recherche d'information [cs.IR]

Alexandru Costan. From Big Data to Fast Data: Efficient Stream Data Management. Distributed, Parallel, and Cluster Computing [cs.DC]. ENS Rennes, 2019. ⟨tel-02059437v1⟩