WSO2 Virtual Hackathon Big Data in the Cloud Case Study
5 likes710 views
This document summarizes WSO2's first virtual hackathon where teams built scalable solutions to process 4 billion real-time events from a smart home device dataset. The teams deployed stream processing and data analytics clusters on the cloud across 5 regions and 100 instances using WSO2 technologies like CEP and DAS. The winning team hosted their scalable architecture on Apache Stratos/WSO2 PaaS using technologies like Kubernetes, CoreOS, Docker and monitoring tools.
WSO2 Virtual Hackathon Big Data in the Cloud Case Study
- 1. WSO2 Virtual Hackathon: Big Data in the Cloud Case Study Lakmal Warusawithana Director - Cloud Architecture - WSO2 Inc Vice President, Apache Stratos
- 2. WSO2 Virtual Hackathon ● In celebration of our 10th year anniversary this year we have conducted WSO2’s first ever virtual hackathon. ● We had ten teams from around the world build a scalable solution that processes up to 4 billion real-time events within a short period of time, all on the cloud.
- 3. Big Data Set ● Used DEBS 2014 Grand Challenge Data set ● The data set originated from 2125 smart plugs deployed across 40 houses ● Each household contained several smartplugs. Each smart plug contained two sensors: ○ A load sensor measuring current load with Watt as unit ○ A work sensor measuring total accumulated work since the start (or reset) with kWh as unit ● 4 Billion Events
- 4. Big Data Challenge ● Query 01: Load Prediction ○ The goal of this query is to make load forecasts based on current load measurements and those of recorded historical data ○ Tool - WSO2 CEP ● Query 02: Outlier ○ The goal of this query is to find devices that have very high data (outlying) readings ○ Tool - WSO2 DAS
- 5. Cloud Challenges ● Hosting scalable WSO2 CEP and DAS deployment in the cloud ● Hosting across 5 EC2 Region, 100 EC2 Instances ● Automating deployment across 100 instances ● Using 300 Docker Containers ● Placing monitoring system on ○ cluster level ○ node level ○ container level ● Auto Healing and recovering
- 7. Technologies used for CEP Cluster ● WSO2 CEP ● Apache Storm ● Nimbus ● Zookeeper
- 8. WSO2 CEP
- 9. WSO2 CEP and Apache Storm
- 11. Distributed WSO2 CEP Deployment on WSO2 PPaaS
- 12. Technologies used for DAS Cluster ● WSO2 DAS ● Apache Spark ● Apache HBase ● HDFS ● Zookeeper
- 13. WSO2 DAS
- 15. Distributed DAS Deployment on WSO2 PPaaS
- 16. Technologies used for Cloud deployment ● Apache Stratos/WSO2 PPaaS ● EC2 ● Kubernetes ● CoreOS ● Flannel
- 17. WSO2 PPaaS
- 18. Kubernetes
- 19. Technologies used for Monitoring system ● Kube-UI ● cAdvisor ● Heapster ● InfluxDB ● Grafana
- 20. Monitoring K8S Cluster Usage - Kube UI
- 21. Monitoring Container’s Resource Usage - cAdvisor
- 22. Monitoring Aggregated Resource Usage - Grafana
- 24. Winning Team!!
- 25. More info... ● http://virtual15.wso2hackathon.com/ ● http://lakmalsview.blogspot.com/2015/10/wso2- virtual-hackathon-big-data-in.html
- 26. Thank You