The Big Data Ecosystem at LinkedIn
Download as PPTX, PDF25 likes8,161 views
The document discusses LinkedIn's approach to building a big data ecosystem. It describes how different data infrastructure systems like Hadoop, Kafka and Voldemort fit together to enable applications like search, recommendations, social graph and analytics. The talk outlines the key paradigms of request/response, streaming and batch processing. It also discusses why LinkedIn chose to build and contribute many open source projects to handle its data infrastructure needs.
The Big Data Ecosystem at LinkedIn
- 1. The Big Data Ecosystem at LinkedInJay Kreps
- 2. MeBackground in data not infrastructureLinkedIn’s SNA teamOriginal co-author of some LinkedIn open source projects (Voldemort, Azkaban, Kafka)
- 3. This TalkWe are in a renaissance of data infrastructure.How do all these pieces fit together?
- 4. Why the current obsession with “Big Data”?
- 5. The goal of modern data infrastructure is to make many small computers act like one big one.
- 9. Infrastructure Icebergs90k lines of tooling and monitoring, 30k lines of logicDedicated engineers, operationsTrainingFirst three nines come from operations
- 10. This is (still) a very immature space. Which systems should we have?
- 11. Infrastructure is sculpted by applications and constraintsProjects are defined by trade-offs
- 12. ConstraintsHardwareJeff Dean: Numbers everyone should knowDavid Patterson: Latency lags bandwidth$$$OtherPath dependenceComplexityResources
- 13. Applications
- 14. Common categories of non-CRUDRecommendations & MatchingGraphsSearchData NormalizationNews feedAnalysis & Monitoring
- 15. Social Graph
- 16. Search
- 21. Analytics
- 22. InfrastructureSearchLuceneBobo (facets), Zoie (real-time indexing), Sensei (distribution)Social GraphStorageOracleVoldemortEspressoStreamsDatabusKafkaOfflineHadoop & friends (Pig, Hive, Azkaban, etc)
- 23. Three Major ParadigmsRequest/ResponseSearchSocial GraphStorageStreamsKafkaBatchHadoop
- 24. Most features are multi-paradigm
- 26. Request/Response PatternsBroker, scatter-gatherStorage systems: only Partitioning strategyLatency oriented
- 27. Batch: HadoopUsesAd hocProduction batchEcosystemHive, PigAzkaban (workflow)Avro dataData in: KafkaData out: Voldemort, Kafka
- 28. Why do batch if you have real-time?Batch advantagesSafetyEasyThroughputSimplicityEconomicsTricky bit: engineering the data cycle
- 29. Why do streaming?You have to glue all these systems togetherThroughput as good as batchLatency much betterMetaphor more natural for low latency than Hadoop
- 30. What makes successful infrastructure systems?Operability and OperationsMonitoringSimplicityDocumentationBroad adoptionLazy usersOpen source
- 31. Open SourceData > InfrastructureOpen source creates better code—even with few outside contributorsCommercial infrastructure not interesting
- 32. Open Source ProjectsWe madeVoldemort: Key/Value storageSensei, Bobo, Zoie: Elastic, faceted, real-time search with LuceneKafka: Persistent, distributed data streamsNorbert: Cluster aware RPC, load balancing, and group membershipAnd others…We stoleHadoop, Pig, HiveLuceneNetty, JettyZookeeperAvroApache Traffic Server
Editor's Notes
- #11: Good news for users, bad news for distributed systems nerdsFilesystems take a decade to mature. Don’t expect this will be easier.