High Throughput Data Analysis
2 likes5,022 views
The document discusses a high-throughput data analysis platform designed to process streaming data efficiently, focusing on a specific case study of resolving virtual machine profiles. Key strategies include continuous data collection, fast processing using a combination of MongoDB and map/reduce techniques, and a modular architecture for real-time analytics. The findings emphasize the importance of quick learning cycles and scalability in data processing to enable timely insights and alerts.
High Throughput Data Analysis
- 1. High-throughput data analysis A Streaming Reports PlatformAuthorsJ Singh, Early Stage ITDavid Zheng, Early Stage ITContributorSatya Gupta, Virsec SystemsOctober 3, 2011
- 2. High-throughput data analysisA few examples of streaming data problemsA concrete problem we solvedHow we solved itTake-away lessons
- 3. Streaming DataData arrives continuouslyMust be processed continuouslyEmit analysis results or alerts as needed
- 7. High-throughput data analysisA few examples of streaming data problemsA concrete problem we solvedHow we solved itTake-away lessons
- 8. Example Use CaseResolve Virtual Machineprofiles application and gathers data about the applicationThe data analysis blocked until data collection was complete, Took several hours before conclusions could be drawnProject goalsStream-mode analysisBegin within a few seconds of start of profilingContinuous updateData rates up to 5 GB per hourAbility to sustain rate for 24 hrsA product of Virsec SystemsAnalysis and Reporting configured to run in the Amazon EC2 environmentCan be scaled up (bigger machines)
- 9. Or scaled out (more machines)ApproachA variant of Eric Ries’ Lean Startup approachIntroduced in his book, The Lean Startup, Crown Publishing Group, 2011It is recipe for learning quicklyRe-adapted by us for this and other “learning projects”Do what it takes to get an end-to-end solutionMeasure, Learn, Build, repeatWhen the cycle has stabilized, expand scope appropriately
- 10. RequirementsFast inserts into the databaseThenature and amount of analysis required was hard to judge in the beginningPrevious experience with Map/Reduce in the Google App Engine environment had shown promise but GAE was not appropriate for this applicationSlick, demo-worthy web interface for presenting resultsStream-mode operationStart showing results within a few seconds of starting the Resolve Virtual Machine, and update it periodically as more data is collected and analyzed.
- 11. High-throughput data analysisA few examples of streaming data problemsA concrete problem we solvedHow we solved itTake-away lessons
- 12. Key decisionsChunk up data into 1-second “slices” as it arrivesUse a collection for signaling the availability of each data sliceProcess each chunk as it becomes availableUse Map/Reduce for analysisExploit Parallelism of the data by using as many processors as needed to maintain the “flow rate”Pipeline the various Map Reduce jobs to maintain sequentiality of data
- 13. Pipeline Component: ListenerListenerGoal: push the data into MongoDB as fast as possibleReceives the data from the Resolve Virtual Machine and stores it into MongoDBSelf-describing data12 different types of data fed over 12 different socketsWritten in C++Socket Interface at one endMongoDB C++ driver at other end
- 15. Did everything it was supposed to do
- 16. Allowed us to focus on our problem, not on MongoDB
- 17. Will use replica sets for making the data available to analysis serversPipeline Component: Map/ReduceAnalysis Program“Function Call Structure” Data TypeCalculation of T was better done in the listener, moved there.Could scale the solution up, but could not scale it outMap Reduce was much faster and could scale out“Memory Usage” Data TypeNeeded multiple map/reduce stages
- 18. “Function Call Structure” AnalysisFnNameTotalTimeSrcFnAddress PIDSF_CFAmap (emit: FnName, TotalTime, min_addr, NumOfCalls, PID, …)Shuffle stage:FnName: CreateRaceObjects { {TotalTime: 3, min_addr: 2 , NumOfCalls: 1 , PID: 1, …} {TotalTime: 7, min_addr: 3, NumOfCalls: 1, PID: 1, …} {TotalTime: 4, min_addr: 1, NumOfCalls: 1, PID: 1, …}{TotalTime: 6, min_addr: 1, NumOfCalls: 1, PID: 1, …} } reduceOutput:{FnName: CreateRaceObjects{TotalTime: 20, min_addr: 1 , NumOfCalls: 2 , PID: 1, …}
- 19. Pipeline Component: PresentationPresentationEach feed type presents a different view of the user’s application.
- 20. And requires a page design of its own
- 21. Tool of choice: DjangoNonRel
- 22. But Python driver for MongoDB was sufficient for most work.
- 23. DjangoNonRel was not really required.High-throughput data analysisA few examples of streaming data problemsA concrete problem we solvedHow we solved itTake-away lessons
- 25. Endpoint StackData Capture (Listener)Custom, preferably written in C++ or JavaNoSQL DatabaseMongoDBWell suited for high speedinsertsCalculation PlatformMongoDB Map/ReduceCould use Hadoop but startup times are a concernPresentationDjango Non-Rel
- 26. About UsInvolved with Map/Reduce and NoSQL technologies on several platformsMany students in J’s Database Systems class at WPI did a project on a NoSQL database.DataThinks.org is a new service of Early Stage ITBuilding and operating “Big Data” analytics servicesThanks