MongoDB Solution for Internet of Things and Big Data
4 likes1,046 views
The document outlines MongoDB solutions for the Internet of Things (IoT) and big data, presenting Zero12 as an innovation company focused on software development for digital transformation. It discusses key concepts like the structure and advantages of MongoDB, especially in handling time series data and various modeling strategies. Additionally, it provides insights into data aggregation, document design, and performance considerations for efficient data operations.
![What’s a document?
{
"name": "John",
"surname": "Doe",
"email": "jdoe@gmail.com",
"cell": 3281432896,
"sport": ["swimming", "football"]
}](https://image.slidesharecdn.com/festivalictlabmongodb-151118154604-lva1-app6891/85/MongoDB-Solution-for-Internet-of-Things-and-Big-Data-15-320.jpg)
![{
_id: ‘ObjectId(“4b2b9…”)’,
first_name: ‘Paul’,
surname: ‘Miller’,
city: ‘London’,
location: [45.123,47.232],
cars: [
{ model: ‘Bentley’,
year: 1973,
value: 100000, … },
{ model: ‘Rolls Royce’,
year: 1965,
value: 330000, … }
]
}
Comparison Relational vs Document](https://image.slidesharecdn.com/festivalictlabmongodb-151118154604-lva1-app6891/85/MongoDB-Solution-for-Internet-of-Things-and-Big-Data-16-320.jpg)
![Document
{
_id: ‘ObjectId(“4b2b9…”)’,
first_name: ‘Paul’,
surname: ‘Miller’,
city: ‘London’,
location: [45.123,47.232],
cars: [
{ model: ‘Bentley’,
year: 1973,
value: 100000, … },
{ model: ‘Rolls Royce’,
year: 1965,
value: 330000, … }
]
}
Example Data Type
Null
Boolean
Number
String
Date
Array
Embedded documents
{ x: null }
{ x: true }
{ x: 3.14 } { x: 3 }
{ x: “zero12” }
{ x: new Date() }
{ x: [“a”,”b”, “c”] }
{ x: {y: “a” } }](https://image.slidesharecdn.com/festivalictlabmongodb-151118154604-lva1-app6891/85/MongoDB-Solution-for-Internet-of-Things-and-Big-Data-17-320.jpg)
MongoDB Solution for Internet of Things and Big Data
- 1. Designing MongoDB solutions for the Internet of Things and Big Data
- 2. INNOVATION COMPANY Roberto Contiero r.contiero@zero12.it @contieroroberto “Learn quickly and Think Well!” Stefano Dindo s.dindo@zero12.it @stefanodindo
- 3. “We are an Innovation Company. We design and develop cutting edge software to drive our customers’ digital transformation, through Agile Methodologies and continuous delivery”
- 4. WE HELP OUR CUSTOMERS TO DESIGN IDEA CREATE PRODUCTS EXTRACT VALUE FROM DATA We get powerful ideas to market fast We design and develop innovative and better software solutions We collect and analyze data to help your decisions
- 5. Discover Experiment Delivery MVP Continuous Design & Integration OUR APPROACH MVP MVP MVP ? ? ? ? ? ? ? End UsersIdeaCustomer & zero12 collaboration meeting ( CanvUX ) Customer feedback
- 6. Today we are dealing with “Internet Of Things”
- 7. “Internet of Things is a neologism referring to the extension of the Internet to the world of objects and concrete places.”
- 8. 2020 IoT Market Share 4 Billion Connected People $4 Trillion Business Opportunity 25+ Billion Integrated systems connected to the Web 50 Trillion 50GBs of data Fonte: IDC
- 9. Broker MQTT Authentication API Business Logic API Predictive Engine API Application Frontend MongoDB IoT Architecture: Users Things Predictive Algorithm Data Operation User Interaction Authentication
- 11. What is MongoDB?
- 13. General Purpose Document Database Open Source MongoDB
- 14. General Info SQL - like MongoDB World Database Database Table Collection Row Document Column Field
- 15. What’s a document? { "name": "John", "surname": "Doe", "email": "jdoe@gmail.com", "cell": 3281432896, "sport": ["swimming", "football"] }
- 16. { _id: ‘ObjectId(“4b2b9…”)’, first_name: ‘Paul’, surname: ‘Miller’, city: ‘London’, location: [45.123,47.232], cars: [ { model: ‘Bentley’, year: 1973, value: 100000, … }, { model: ‘Rolls Royce’, year: 1965, value: 330000, … } ] } Comparison Relational vs Document
- 17. Document { _id: ‘ObjectId(“4b2b9…”)’, first_name: ‘Paul’, surname: ‘Miller’, city: ‘London’, location: [45.123,47.232], cars: [ { model: ‘Bentley’, year: 1973, value: 100000, … }, { model: ‘Rolls Royce’, year: 1965, value: 330000, … } ] } Example Data Type Null Boolean Number String Date Array Embedded documents { x: null } { x: true } { x: 3.14 } { x: 3 } { x: “zero12” } { x: new Date() } { x: [“a”,”b”, “c”] } { x: {y: “a” } }
- 18. Aggregation op1 op2 opn…… { "name": "John", "surname": "Doe", "email": "jdoe@gmail.com" } Pipeline stages Documents
- 20. Main Operator • $project • $match • $limit • $skip • $sort • $unwind • $group
- 21. Join ?
- 23. MongoDB 3.2 MongoDB Compass 3 Storage Engine • WiredTiger • NMAPv1.1 • In-Memory ( beta ) Data Encryption Business Intelligence Connectors $lookup operator Document Validation
- 25. Node 1 Secondary Node 2 Secondary Node 3 Primary Heartbeat Replica Replica ReplicaSet Configuration
- 28. How to Use MongoDB in IoT Area Time Series
- 29. Definition Set of values of a variable detected at different timestamps. Time t0 t1 t2 t3 f ( t0 ) f ( t1 ) f ( t2 ) f ( t3 )
- 30. Time Series Data is Everywhere 1. Financial markets pricing 2. Sensors (temperature, pressure, proximity) 3. Industrial Fleets (Location, velocity, operational) 4. Social Networks (status update) 5. System (server logs, application logs) 6. Mobile devices (calls, texts)
- 31. Time Series Data at a Higher Level 1. Widely applicable data model 2. Various schema and modeling options 3. Application requirements drive schema design
- 32. Time Series - Schema Design How to Use MongoDB in IoT Area
- 33. Designing for writing and reading 1. One document per event 2. One document per minute (average) 3. One document per minute (second) 4. One document per hour
- 34. One document per event { server: "server1", load: 92, ts: ISODate("2014-10-16T22:07:38.000-0500") } 1. Relational-centric approach 2. Insert-driven workload 3. Aggregations computed at application-level
- 35. One document per minute (average) { server: "server1", load_num: 92, load_sum: 4500, ts: ISODate("2014-10-16T22:07:00.000-0500") } 1. Pre-aggregation to compute average per minute more easily 2. Update-driven workload 3. Minute-level resolution
- 36. One document per minute ( second ) { server: "server1", load: { 0: 15, 1: 20, ..., 58: 45, 59: 40 } ts: ISODate("2014-10-16T22:07:00.000-0500") } 1. Store per second data at minute level 2. Update-driven workload 3. Pre-allocate structure to avoid document moves
- 37. One document per hour ( by second ) { server: "server1", load: { 0: 15, 1: 20, ..., 3598: 45, 3599: 40 } ts: ISODate("2014-10-16T22:00:00.000-0500") } 1. Store per second data at hourly level 2. Update driven workload 3. Pre-allocate structure to avoid document moves 4. Updating the last second requires 3599 steps
- 38. One document per hour ( by second ) { server: "server1", load: { 0: {0: 15, ..., 59: 45}, .... 59: {0: 25, ..., 59: 75} } ts: ISODate("2014-10-16T22:00:00.000-0500") } 1. Store per second data at hourly level with nesting 2. Update-driven workload 3. Pre-allocate structure to avoid document moves 4. Updating the last second requires 59+59 steps
- 39. Writing operation analysis 1. Example: data generated every second 2. Capturing data per minute requires: - One document per event: 60 writes - One document per minute: 1 write, 59 updates 3. Transition from “insert-driven” to “update-driven” - Individual writes are smaller - Performance and concurrency benefits
- 40. 1. Example: data generated every second 2. Reading data for a single hours requires: - One document per event: 3600 reads - One document per minute: 60 reads 3. Read performance is greatly improved: - Fewer disk seeks - Optimization with tuned block sizes and read ahead Read operation analysis
- 41. Live Demo http://www.zero12.it/lab-festivalict/ E’ possibile scaricare semplici esercizi realizzati per il Lab da: