Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark Meetup
5 likes1,606 views
The document outlines a presentation by Jonathan Winandy on building data labs that transform organizations using existing data, emphasizing that data itself has no intrinsic value. It discusses the creation of data platforms using big data technologies, ETL workflows, and the importance of storage and accessibility for business views. It also presents strategies for developing a data lab incrementally with business-oriented use cases while acknowledging the challenges and investment concerns inherent in such projects.
![Rex > Datalab > CoGroup
{
"group":123,
"V":[{"c2":true,
"c1":123}],
"R":[{"c3":"DIRECT",
"c2":"boeuf bourguignon",
"c1":123},
{“c3":"DIRECT",
"c2":"nouilles de riz",
“c1":123},
{“c3":"INDIRECT",
"c2":"soupe au melon d’hiver",
"c1":123},
{"c3":"INDIRECT",
"c2":"nouilles de riz",
“c1":123}]}
}
group int
v array<struct<c1:int,
c2:boolean>>
r array<struct<c1:int,
c2:string,
c3:string>>](https://image.slidesharecdn.com/datalab101-151207191005-lva1-app6891/85/Datalab-101-Hadoop-Spark-ElasticSearch-par-Jonathan-Winandy-Paris-Spark-Meetup-20-320.jpg)
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark Meetup
- 1. U n i v a l e n c e DATALAB 101 Jonathan WINANDY
- 2. About me Jonathan WINANDY Data Engineer / Entrepreneur @AHOY_JON U n i v a l e n c e
- 3. Présentation What are Datalabs ? Projects to transform an organisation based on its existing data.
- 4. Présentation Why ? Data is a leverage for economic growth.
- 5. Présentation But ? Data has no value by itself.
- 6. Is data the new oil ?
- 7. Présentation How do we start ? By building a Data platform ?
- 8. Présentation Data Platform Awesome pipelines + BIG Data technologies
- 9. Rex > Data Platform U n i v a l e n c e Rex > Data Platform > Schéma cible Staging DWH Business Views sql3 sql2 sql1 Logs Events other cube sql Serving Metadata
- 10. Rex > Data Platform U n i v a l e n c e Rex > Data Platform > Schéma cible Staging DWH Business Views sql3 sql2 sql1 Logs Events other cube sql Serving Metadata Staging : Storage space used to decouple from upstream sources.
- 11. Rex > Data Platform HADOOP ETL workflow : Rex > Data Platform > Data Warehouse > ETL API 1 (file) API 2 (file) Ref (file)DB API adapter result DB adapter DB adapter serving DBFilesFilesFiles processprocessprocess U n i v a l e n c e
- 12. Rex > Data Platform U n i v a l e n c e Rex > Data Platform > Business views & Reporting ● Création des axes métiers ● Visualisation des données DWH BV BV BV DB SQL Self service Data visualisation
- 13. Rex > Data Platform Objectives : Storage / Warehousing. Reduce access time. Elasticity. Collaboration. Reuse. U n i v a l e n c e
- 14. Présentation But ? Building a data platform is a BIG project with no clear return on investment.
- 15. Présentation “The Datalab as an infrastructure.”
- 16. Présentation How to grow a Datalab ? Start small with an end to end business case.
- 17. Rex > Datalab U n i v a l e n c e
- 18. U n i v a l e n c e CoGroup Map
- 19. Rex > Datalab > Recipe 1. Stage the data 2. Source mapping 3. CoGroup 4. Enrich 5. Make it accessible Sprint A. Cardinality Study B. Technical mapping C. Business-oriented model Marathon
- 20. Rex > Datalab > CoGroup { "group":123, "V":[{"c2":true, "c1":123}], "R":[{"c3":"DIRECT", "c2":"boeuf bourguignon", "c1":123}, {“c3":"DIRECT", "c2":"nouilles de riz", “c1":123}, {“c3":"INDIRECT", "c2":"soupe au melon d’hiver", "c1":123}, {"c3":"INDIRECT", "c2":"nouilles de riz", “c1":123}]} } group int v array<struct<c1:int, c2:boolean>> r array<struct<c1:int, c2:string, c3:string>>
- 21. Rex > Datalab > Ex f: G => Visiteur
- 22. Rex > Datalab > Query select count(*) from visitor, visitor.session session, session.page page where visitor.is_robot = false and page.type = product U n i v a l e n c e Query for nested Data (Impala) :
- 23. Rex > Datalab > Sum UP CoGroup all your inputs with PIG. Map the data with Spark. Store in ElasticSearch.