The State of BigData - meetup bigdata @ovh
- 1. @PennAnData
- 2. The State of Big Data 2016
- 3. Summary 1 Data Facts 2 Hadoop Basics 3 Beyond Batch : Streaming 4 Columnar Storage 5 Ecosystem
- 4. Big Data Facts PART 1
- 6. Volume ... data Production constantly growing data Retention increase widely Extract Value from you data Storage Cost decrease
- 7. Velocity ... Data produced Faster Get Real Time insight Move from capture to Analysis Get Actionable insight
- 8. Variety ... Not only Structured Data Toward mostly Unstructured Text (articles, comments, tweets,...) Images (id cards, bills,...) Logs, metrics,...
- 10. Seek Time • 5-10ms • 200 move/s
- 11. Data Transfer Rate Mbps 100 1000 10000 MB/s 12.5 125 1250 1 Mo 80ms 8ms 0.8ms 1 CD (700 Mo) 56s 5.6s 0.56s 1 Go (1000 Mo) 1m20 8s 0.8s 1 DVD (4700 Mo) 6m16 37.6s 3.76s 1 To (1000 Go) 22h13 2h13m 13m
- 12. Data Transfer Rate Mbps 100 1000 10000 MB/s 12.5 125 1250 1 min 750 MB 7.5 GB 75 GB 15 min 11 GB 112 GB 1 TB 1 hour 45 GB 450 GB 4.5 TB 1 day 1TB 10.8 TB 108 TB
- 14. Payload
- 15. Definition “Big Data really is about having insights and making an impact on your business. If you aren’t taking advantage of the data you’re collecting, then you just have a pile of data, you don’t have Big Data.” #BigData
- 19. #DougCutting
- 20. #Tools
- 21. Timeline
- 22. #HDFS
- 23. #Blocks HDFS
- 25. File Blocks DataNodes / HDFS / Replication
- 26. DataNodes NameNode / HDFS / NameNode
- 27. DataNodes NameNodes / HDFS / Namespace #Federation
- 28. / HDFS / HA #HighAvailability NN1 NN2
- 29. / HDFS / HA Failover Controller ● NameNode Side ● Health monitor ● Manage HA State ● Zookeeper Side ● Monitor State ● Maintain or Try to get Active Lock #Five9rulez
- 30. / HDFS / Client #Read #DataLocality
- 31. / HDFS / Client #Write #ReplicationFactor3
- 32. #MapReduce
- 33. MapReduce #MAP
- 36. MapReduce <key1, val1> map <key2, val2> map reduce <okey1, oval1> reduce <okey2, oval2> <key3, val3> map <key500, val500> map <ikey2, ival521><key501, val501> map reduce <okey150, oval150> <key502, val502> map <ikey150, ival522> <ikey1, ival1> <ikey2, ival2> <ikey1, ival3> <ikey2, ival4> <ikey150, ival520> Input Input Pairs Intermediate Pairs Output Pairs Output Step 1: Split ) Step 2: Map Step 3: Shuffle / Sort Step 4: Reduce Step 5: Store )
- 37. MapReduce
- 38. MapReduce
- 39. #Pig & #Hive
- 40. Hive ● Tez ● Impala ● Presto.io
- 41. #HBase
- 42. HBase #Model
- 43. HBase #Model
- 44. HBase #Model
- 45. HBase #Model
- 47. HBase #Scale
- 48. HBase #Scale
- 49. HBase #Meta
- 50. #HBaseArch
- 51. HBase #SQL
- 52. HBase #Features ● Coprocessor ● Auto-sharding ● Scan (full,range) ● Schemaless ● Cell versioning ● Battle tested ● Compactions ● Replications ● Custom filters ● Transactional ● Low Latency ● Active Community
- 53. Beyond Batch : Streaming PART 3
- 54. / Streaming / Data Platform #Transport
- 55. / Streaming / Data Platform / Kafka
- 56. + = / Streaming / Frameworks
- 57. / Streaming / Storm / Topology #Storm
- 58. / Streaming / Storm / Topology #Parallelism
- 59. / Streaming / Flink #Job
- 60. / Streaming / Flink #DataSet API #DataStream API
- 61. Ok Steven, but a new DSL for each new hype tool ? Come on...
- 64. Apache Beam #Features ● Open Sourced Google DataFlow ● Unify bigdata developements ● Beam Model (from DataFlow model) ● Parallel Data processing Pipelines ● Pluggable runners: Flink or G Cloud DataFlow ● Portability ● SDKs : Java / Python
- 65. #Architecture
- 67. Drawbacks • Hard to merge for serving layer • Hard to maintain and operate both realtime and batch code in sync
- 69. From Storm to Flink
- 70. #Yarn
- 71. Yarn #MapReduce
- 76. Yarn Frameworks
- 77. #Mesos
- 81. Ecosystem PART 5
- 83. Vendors
- 84. Integration