MongoDB World 2019: Exploring your MongoDB Data with Pirates (R) and Snakes (Python)

Ken W. Alger – Developer Advocate, MongoDB
Exploring your MongoDB Data with Pirates (R) and
Snakes (Python)
@kenwalger

Ken W. Alger
Developer Advocate

Overview
§ The Document Model
§ Data Frames
§ R vs. Python
§ MongoDB to Data Frames
§ Array Consumption
§ The Power of MongoDB

Document Model Features
Naturally maps objects to
code using JSON
Represent data of any
structure. Our data model is
very flexible.
Strongly typed for ease of
processing. We support
over twenty binary
encoded JSON data types.

Document Model for Analytics
Flexibility helps with
feature engineering by
allowing for
experimentation and the
picking of features
iteratively.
For Deep Learning the
flexibility allows for faster
iteration.
Pre-filtering of data with
aggregation framework.

Flexibility is Great if You're a Python or a Pirate…

Sail Data
{
"Name": "Götheborg",
"Year Completed": 1738,
"Sail Area": {
"Lateen mizzen": 160,
"Mizzen topsail": 2500,
…
},
…
}
{
"Name": "Batavia",
"Year Completed": 1628,
"Sail Area": {
”Main topsail": 3500,
…
},
…
}

…but some tasks require data that's rigidly structured.

Data Frame
A data frame is a list of vectors, factors,
and/or matrices all having the same length
(number of rows in the case of matrices).
Used for storing data tables.

Data Frame Data
Length (ft) Year Completed Displacement Sail Area (sq. ft)
Batavia 186 1628 1200 33000
Cutty Sark 280 1869 2100 32000
Götheborg 190 1738 NaN 21140
HMS Endeavor 97.75 1764 NaN 29889
Kruzenshtern 375 1926 3064 NaN
HMS Victory 227.5 1765 3500 58556

Distributed Data Frames
Big data distributed across clusters.
Use tools like:

R Data Frame
R dataframe is more or less built into the
language.
More functional than Python.
More statistical support in general.

Python Data Frame
More object-oriented.
Relies on packages (pandas, numpy, scikit-
learn)
As a language it’s great for additional tasks
along side of analytics.

Sail Data
{
"Mizzen topsail": 400,
"Mainsail": 2500,
…
}
{
"Main topgallant": 600,
"Mainsail": 3500,
…
}

library(mongolite)
connection <-
mongo(collection = "sails",
db="ships",
url="mongodb://localhost”)
sails <- data.frame(
connection$find()
)
MongoDB Data to Data Frames
import pandas as pd
from pymongo import
MongoClient
client =
MongoClient('localhost',
27017)
db = client.ships
df = pd.DataFrame(list
(db.sails.find())
)

Data Frame
Lateen mizzen Main topgallant Mainsail Mizzen topsail _id
0 120.0 NaN 2500 400.0 5cf53129984ae2b600701611
1 120.0 600.0 3500 NaN 5cf53141984ae2b600701612
Lateen mizzen Mizzen topsail Mainsail Main topgallant
1 120 400 2500 NA
2 120 NA 3500 600

Array Data
Pattern 1 – Arrays of Arrays
sail_area = [
[160, 2500, 1600, 450],
[250, 3500, 2800, 575],
[120, 3500, 295]
]
0 1 2 3
0 160 2500 1600 450
1 250 3500 2800 575
2 120 3500 295 NaN
Resulting Data Frame
pd.DataFrame(sail_area)
-or-
data.frame(sail_area)
pd.DataFrame(list(db.sail_area.find()))
-or-
data.frame(connection$find())

Array Data
Pattern 2
[
{"lateen mizzen": 160, "mail topsail": 2500, "mainsail": 1600, "main topgallant": 450},
{"lateen mizzen": 250, "mail topsail": 3500, "mainsail": 2800, "main topgallant": 575},
{"lateen mizzen": 120, "mailsail": 3500, "jib": 450},
]
_id jib lateen mizzen mail topsail mainsail main topgallant
0 5cf5607b984ae2b600701613 NaN 160.0 2500.0 1600.0 450.0
1 5cf5609f984ae2b600701614 NaN 250.0 3500.0 2800.0 575.0
2 5cf560d4984ae2b600701615 450.0 120.0 NaN 3500.0 NaN

Array Data
Pattern 3
[
{"area": [160, 2500, 1600, 450]},
{"area": [250, 3500, 2800, 575]},
{"area": [120, 3500, 295]}
]
_id area
0 5cf57574984ae2b600701623 [160.0, 2500.0, 1600.0, 450.0]
1 5cf57584984ae2b600701624 [250.0, 3500.0, 2800.0, 575.0]
2 5cf5758f984ae2b600701625 [120.0, 3500.0, 295.0]

Moving Data from MongoDB Arrays

Array Data
[
{"name": "Batavia", "area": [160, 2500, 1600, 450]},
{"name": "Götheberg", "area": [250, 3500, 2800, 575]},
{"name": "HMS Endeavor", "area": [120, 3500, 295]}
]

library(mongolite)
connection <- mongo(collection =
"sails", db="ships",
sails <-
Working with MongoDB Arrays
import pandas as pd
from pymongo import MongoClient
client = MongoClient('localhost',
27017)
db = client.ships
values = []
for ship in db.sailareas.find():
values.append(ship['area'])
print(pd.DataFrame(values))

Array Data
[
]
0 1 2 3
0 160.0 2500.0 1600.0 450.0
1 250.0 3500.0 2800.0 575.0
2 120.0 3500.0 95.0 NaN

library(mongolite)
connection <- mongo(collection =
"sails", db="ships",
sails <-
Working with MongoDB Arrays
import pandas as pd
from pymongo import MongoClient
client = MongoClient('localhost',
27017)
db = client.ships
values = []
seriesLabels = []
for ship in db.sailareas.find():
values.append(ship['area'])
seriesLabels.append(ship['name'])
print(pd.DataFrame(values,
index=seriesLabels))

Array Data
[
]
0 1 2 3
Batavia 160.0 2500.0 1600.0 450.0
Götheborg 250.0 3500.0 2800.0 575.0
HMS Endeavor 120.0 3500.0 95.0 NaN

The Power of
MongoDB
Aggregation Framework

Aggregation Framework
• Pre-filter and/or pre-aggregate data on the server before
moving it across the network.
• Reduces the amount of data in the data frame.
• Improves performance.

Sample Data
Country
Year
Completed
Displacement
Individual Sail Areas
(sq. ft)
Batavia NLD 1628 1200 [292, 2012, 990, 550, 403, 642, 1056, ...]
Cutty Sark GBR 1869 2100 [2408, 866, 155, 2041, 518, 1675, …]
Götheborg SWE 1738 NaN [315, 614, 314, 2451, 2096, 2477, …]
HMS
Endeavor
GBR 1764 NaN [1060, 2089, 1101, 420, 2320, 2245]
Kruzenshtern DEU 1926 3064 [1476, 1352, 2383, 1100, 1807, 448, 2415]
HMS Victory GBR 1765 3500 [1310, 2445, 1327, 1668, 2098, 2179, …]

from datetime import datetime, timezone
values = []
seriesLabels = []
for ship in db.ships.aggregate [
{
'$match': {
'year_completed': {
'$gte': datetime(1571, 1, 1, 0, 0, 0, tzinfo=timezone.utc)
}
}
},
Aggregation Pipeline 1

{
'$match': {
'year_completed': {
'$lt': datetime(1862, 1, 1, 0, 0, 0, tzinfo=timezone.utc)
}
}
}, {
'$match': {
'country_of_origin': {
'$ne': 'USA'
}
}
},

{
'$project': {
'name': 1,
'country_of_origin': 1,
'sail_areas': 1,
'total_sails': {
'$cond': {
'if': {
'$isArray': '$sail_areas'
},
'then': {
'$size': '$sail_areas'
},
'else': 'NA'
}
}}},

{
'$project': {
'name': 1,
'country_of_origin': 1,
'total_sails': 1,
'total_area': {
'$sum': '$sail_area'
}
}
}
]):
values.append(ship['sail_area'])
seriesLabels.append(ship['name'])
dataframe = pd.DataFrame(values, index=seriesLabels)

Results
0
La Amistad 19335
Batavia 31246
Götheborg 18464
HMS Endeavour 9235
Golden Hind 11749
Grand Turk 8405
Kalmar Nyckel 3710
Lady Nelson 24938
Pallada 20785
Shtandart 9363
HMS Sultana 35061
HMS Surprise 26272
HMS Trincomalee 21744
HMS Victory 14740

Other Sessions
Today
1:00pm Real-time Clinical Decision Support System – Prem Timisina & Arash Kia
2:00 Analytics with MongoDB – Stuart Shiell & Mark Clancy
2:00 A Complete Methodology to Data Modeling for MongoDB –Daniel Coupal
3:15 Unleash the Power of the MongoDB Aggregation Framework – Abhishek Bagga
Tomorrow
9:00am Best Practices for Working with IoT and Time-series Data – Robert Walters
3:00pm MongoDB in Data Science – Vigen Sahakyan

Takeaways
MongoDB's flexible data model is very powerful for data analytics.
Some analytic tools require a more structured approach.
When forming your data the schema design used can make a huge
impact on analytics.
Use MongoDB's Aggregation Framework to improve performance.

Thank You!
Ken W. Alger - @kenwalger

MongoDB World 2019: Exploring your MongoDB Data with Pirates (R) and Snakes (Python)

MongoDB World 2019: Exploring your MongoDB Data with Pirates (R) and Snakes (Python)

More Related Content

What's hot (20)

Similar to MongoDB World 2019: Exploring your MongoDB Data with Pirates (R) and Snakes (Python) (20)

More from MongoDB (20)

Recently uploaded (20)

MongoDB World 2019: Exploring your MongoDB Data with Pirates (R) and Snakes (Python)