Bigdata analytics and our IoT gateway
Download as PPTX, PDF1 like548 views
The document outlines a presentation on big data analytics in relation to the IoT gateway 'Toon' by Quby, detailing their journey from a few customers to handling data from over a million customers. It describes the architecture, data collection methods, and services offered, including real-time energy consumption analytics and appliance detection. Additionally, the anonymity of customer data and potential services derived from the collected data, such as benchmarking and behavioral studies, are discussed.
Bigdata analytics and our IoT gateway
- 1. 30 March, 2017 Aemro Amare, Freek van Gool 1 Bigdata Analytics and our IoT Gateway :Toon
- 2. Agenda • A little bit about us • Who is Quby & what is Toon? • Our big data journey • Data collection & services • Architecture & technologies • Demo • Questions @javafreekNL @twoxey
- 3. A little bit about us FreekAemro @javafreekNL @twoxey
- 4. First attempt Home control box @javafreekNL @twoxey
- 7. Meter Adapter Gas Sensor Electricity Sensor Z-Wave Phillips Hue WiFi Router Z-wave Smart plugand smoke detectors Central heating boiler Boiler adaptor Toon® Toon connections - Data sources
- 8. The service center Service Center Toon Displays Mobile devices Product Applications Back Office Applications Mobile Backend Internet Internet VPN Access Data collector
- 10. Exponential growth 2013 30.000 2014 90.000 2015 190.000 2016 300.000 2017 (1.000.000) 2013 2014 2015 2016 2017
- 11. Our journey towards bigdata 30 March, 2017 12 2014 • 30K customers • 40 types of sensor data from 3K customers • up to 1 minutes resolution 2012 • 4K customers • Aggregated Energy data collected(DAY,WE EK,MONTH) • Benchmarking functionality for customers 2013 • 90K customers • Data collection from 30K customers • Event and Measurement data
- 12. Steps moving towards bigdata(cont’d) 30 March, 2017 13 2017 • 300K customers • Near Real Time aggregation with segmentation • 400 signal/5 minutes • Sharing Measurement data with tenants 2015 • 190K customers • Apache storm for Near Real Time Energy consumption (3K customers) • Data analysis using Cloudera stack • Up to 10 seconds resolution 2016 • 330K customers • Collect data from 230K customers anonymously • Data as a service • Analytics platform as a service
- 13. Toon® Types of data More than 300types of sensor and user interaction data. If you want to know more please contact us : dteam@quby.com 14 Energy CH-boiler (OT) Thermostat Gas consumption Electricity consumption Water temperature Service messages Burner information Room temperature Thermostat program Manual settings Plug-data Electricity consumption On/off Water pressure Energy production Energy feed-in Boiler status Touch events Thermostat Mobile 1 2 3 4 5 Thermostat setting Hue data
- 14. Anonymous Data Collection 30 March, 2017 15 Service CenterCustomer Always anonymous data collected Customer may opt in for services and send keys Customer may opt out and change keys Data can’t be de-anonymized again
- 15. Sample Services with Toon’s Data? 30 March, 2017 16 1. Benchmarking with daily energy meter data
- 16. Sample Services with Toon’s Data? ( cont’d) 30 March, 2017 17 2. Nearly Real Time Forecasting : Energy data Aggregation
- 17. Sample Services with Toon’s Data? ( cont’d) 30 March, 2017 18 3. Home Appliances Energy Consumption Detection: Energy Data disaggregation
- 18. Sample Services with Toon’s Data ?(cont’d) 30 March, 2017 19 4. Boiler Maintenance Prediction (KetelIQ)
- 19. Sample Services with Toon’s Data? ( cont’d) 30 March, 2017 20 5. Customer’s Behavioral studies Click data Thermostat programs and co2 emission
- 20. What are we collecting? 24K measurements / day / customer 600KB (zipped) 230K customers 21
- 21. Arch. Principle : lambda Arch. 22
- 22. Messaging Data warehouse Tools and Technologies 30 March, 2017 23 Analytic Clusters Tools and Frameworks Containers & micro services Monitoringandalert
- 23. Data flow (partial view) 24
- 24. Things to mention • ssl mutual authentication • raw-data stored partitioned in better way • Customers can control access to their own data 30 March, 2017 27
- 25. What are we doing with all that data? • Firehose – data as a service for B2B (and researchers) • Analytic platform as a service • New data driven services for Toon users • Improve existing services • Marketing strategy (understand your customers) 30 March, 2017 29
Editor's Notes
- #4: Freek van Gool – Cloud Platform Architect We are in talks with many companies and people who have a broader perspective than us on the what is happening in the IoT world. We were told and noticed ourselves we’re doing something remarkable. We want to share our learnings and pitfalls with a larger audience. We don’t have all the answers and solutions and see the IoT field is still changing very quickly. But by sharing our knowledge and experiences we hope we can help more people being active in the IoT industry. Finally we hope we can also learn from your experiences and solutions.
- #5: About Quby Started in 2004 in Amsterdam as Home Automation Europe BV Currently a company of 150 people in Amsterdam with more than 25 nationalities Focus on smart energy & smart living We have built out the whole stack to bring you Toon, from hardware to cloud Smart home before smart home IoT before IoT
- #6: Difficult for end users
- #7: No engagement
- #8: Smart thermostat and IoT gateway
- #9: Consumers currently use Toon® with…. Smart thermostat Smart home hub (solar monitor, mazout, smart plugs, smoke detector) So… How do you sell this as a small company?
- #10: The solution which started the success is this.. (explain the logical setup and users in the solution without diving into all the details)
- #11: The second pivot: how to find the right business model Via energy utilities Help utilities make the energy transition From utility to energy service provider
- #12: We’ve celebrated numerous big successes which were dwarfed already by the next big success in a short time. You can imagine It’s very exciting to be part of a company which experiences exponential growth. However, it also required us to change a lot of things in our organization in order to cope with the growth we experienced.
- #13: ##Number of customers derived from database SELECT YEAR(complete.tijdstip) as jaar, COUNT(*) as number_of_activations FROM ( SELECT STARTDATE as tijdstip FROM quby.DEVICE_PARAMETER_VALUE_HISTORY asstat_hist where value = 'CONNECTED_TO_CLIENT' union SELECT TIMETAKEN as tijdstip FROM quby.DEVICE_PARAMETER_VALUE as stat where value = 'CONNECTED_TO_CLIENT' )as complete group by YEAR(complete.tijdstip) order by jaar desc