Real-time Big Data at FPT (for TechCamp University)
5 likes1,145 views
The document discusses the evolution and significance of real-time big data platforms, emphasizing the transition from traditional big data to fast data since 2015. It presents key open-source technologies such as Apache Spark, Kafka, and RFX for building effective data processing systems. Notably, it shares a user story illustrating the real-time analysis and querying of data to address business needs.
Real-time Big Data at FPT (for TechCamp University)
- 1. Real-time Big Data at FPT and some key ideas to build real-time big data platform from open source tools ○ Apache Spark ○ Reactive Function X (RFX) Presented by @tantrieuf31 http://nguyentantrieu.info
- 2. about me ? ● Full Stack Engineer and Tech Lead at AdsPlay, startup project from FPT Telecom ● Founder at RFXLab.com, building RFX framework and Fast Data Intelligence Platform for Data-driven Organization ● Tech Blogger at http://engineering.adsplay.net
- 3. Abstract 1. Just 5 minutes about the history of “Big Data” 2. Does Big Data solve big problems ? 3. Overview about Open Source Tools a. Netty (Event Collector) b. Kafka (Event Queue) c. RFX-Stream (Event Processor) d. Apache Spark (Big Data processing engine) e. RFX-Iris (Fast Data Query Interface)
- 4. 5 minutes about the history of “Big Data”
- 5. Imagine what if you have to build a GREAT pyramid ? In fact, the Big Data was born in 3000 years ago. When you have to build a great thing, you would face with making decisions with lots of data.
- 8. OK, let’s get back to 2015
- 9. What if the business is not driven by data? Refer: http://www.nytimes.com/2011/04/24/business/24unboxed.html
- 10. Since 2015, the Fast Data, a new trend, has been replacing Big Data http://www.tibco.com/blog/2015/03/27/how-analytics- facilitates-fast-data
- 11. 1970s 1990s 2000s 2010s Data Management Technology and Trends ● Netty.io ● Apache Storm ● Apache Kafka ● Apache Spark ● RFX ● ... ● Hadoop Ecosystem ● NoSQL Ecosystem ● ... ● Oracle ● MySQL ● PostgreSQL ● ...
- 12. “ Does Big Data solve our big problems ?
- 13. tracking all access logs and user’s activities Processing in real- time( seconds) ! Storing multiple types of log (video, web, mobile, like, comment, play, … )
- 17. boosting Sale Revenue / Profit Log events Reactive events
- 18. How is the Big Data used at FPT ?
- 19. Does Vietnamese love football ? The correlation said YES
- 20. Analyzing trending events in real-time !
- 21. Visualizing all user’s devices
- 22. Real-time Big Data Architecture
- 23. “ How to build an “Just-Work” real- time big data system ?
- 24. KEY IDEA is “divide and conquer”
- 25. User Story in plain English 1. Hercules is thinking about some questions. E. g: What’s hot songs of Nhacso on Facebook ? 2. He decides to ask Iris about this question. 3. Iris analyzes the question into “query messages” and deliver them to Zeus. 4. Zeus uses his power of “large-scale data processing” to answer the question. 5. Done, Zeus return the result “hot songs on Facebook” for Iris. 6. She sends the result to Hercules
- 26. Visualizing our user story Question about Big Data: What’s hot songs of NhacSo.net on Facebook ? messages ZeusIris Hercules
- 28. Let’s see how it works
- 29. Awesome Open Source Projects to follow RFXLab.com ◎ http://www.rfxlab.com ◎ https://github.com/rfxlab Kafka : http://kafka.apache.org Hadoop http://hadoop.apache.org Apache Spark https://spark.apache.org
- 30. Awesome Open Source Projects to follow Native Kafka driver: https://github. com/edenhill/librdkafka/ PHP Kafka driver: https://github. com/EVODelavega/phpkafka Data Visualization JavaScript Library https://github.com/nvd3-community/nvd3
- 31. Good ref books "Spend some time alone and learn to develop your personal resources." Alexander Reid Martin