SlideShare a Scribd company logo
Lukáš Korous
lukas.korous@softica.eu
Document database MongoDB
introduction & mobile app backend use
case
Lukáš Korous
Lukáš Korous
lukas.korous@softica.eu
Contents
Introduction ......................................................................................................................................5
Characteristics...........................................................................................................................................5
NoSQL pitfalls:...........................................................................................................................................6
E-shop, Online banking, Stocks trading, … ($$$) ..................................................................................6
Business Intelligence, Reporting, Analysis, DWH, DM..........................................................................6
SQL <-> MongoDB.....................................................................................................................................7
Structures..............................................................................................................................................7
SQL ........................................................................................................................................................7
MongoDB ..............................................................................................................................................7
Select query ..........................................................................................................................................7
SQL ........................................................................................................................................................7
MongoDB ..............................................................................................................................................7
Simple update (invariant to current values).........................................................................................7
SQL ........................................................................................................................................................7
MongoDB ..............................................................................................................................................7
Update using old values........................................................................................................................8
SQL ........................................................................................................................................................8
MongoDB ..............................................................................................................................................8
Authentication in MongoDB .............................................................................................................8
Authenticate a User ..................................................................................................................................8
Centralized User Data ...............................................................................................................................8
Enable Access Control...............................................................................................................................9
Roles..........................................................................................................................................................9
Privileges...................................................................................................................................................9
Inherited Privileges ...............................................................................................................................9
View Role’s Privileges............................................................................................................................9
Users and Roles.........................................................................................................................................9
High Availability (HA) using MongoDB............................................................................................10
Replica Set introduction..........................................................................................................................10
Replica Set – Failure of a Secondary node..............................................................................................10
Lukáš Korous
lukas.korous@softica.eu
Recovery process ................................................................................................................................11
Replica Set – failure of a Primary node...................................................................................................12
Quorum-based voting for replica control [source: en.wikipedia.org]................................................12
Oplog replication lag...............................................................................................................................13
Replica Set – conclusion..........................................................................................................................14
Use case #1: Stand-alone (non-replicated) MongoDB database backend......................................15
Installation ..............................................................................................................................................15
Get the necessary packages................................................................................................................15
Log and data files location setup ........................................................................................................15
Starting the service .............................................................................................................................15
DB Setup..................................................................................................................................................16
DB Operations.........................................................................................................................................16
Insert query – “insert” ........................................................................................................................16
Select query – “find”...........................................................................................................................17
Update query – “update”....................................................................................................................17
Use case #2: MongoDB Replica Set as a mobile app backend........................................................17
Mobile app description...........................................................................................................................17
Technologies used...................................................................................................................................17
Backend network overview ....................................................................................................................18
MongoDB config file................................................................................................................................18
Querying replica set state on a node......................................................................................................19
Schemas used..........................................................................................................................................22
Schemas in the code ...........................................................................................................................22
ReST API on top of MongoDB .................................................................................................................26
Monitoring ..............................................................................................................................................30
Zabbix (open source tool) for MongoDB monitoring..........................................................................30
Quantities monitorable with Zabbix ...................................................................................................30
Services running on the host ..............................................................................................................32
Ports on which MongoDB and Zabbix listen.......................................................................................34
MongoDB logfile - /var/log/mongodb/mongodb.log .........................................................................34
Sources............................................................................................................................................35
Lukáš Korous
lukas.korous@softica.eu
Lukáš Korous
lukas.korous@softica.eu
Introduction
Characteristics
MongoDB is a cross-platform document-oriented database.
Storage pattern is a collection of JSON-like documents with dynamic schemas (MongoDB calls the
format BSON – binary JSON – and it is a binary format), making the integration of data in certain
types of applications easier and faster. Here is an example of a record:
Released under a combination of the GNU Affero General Public License and the Apache License,
MongoDB is free and open-source software.
MongoDB is the most popular NoSQL database in the world:
Lukáš Korous
lukas.korous@softica.eu
SQL vs. NoSQL
SQL (ACID)
 Atomicity: all-or-nothing (transactional processing)
 Consistency: all-the-time? yes
 Isolation: discrete (vs. continuous)
 Durability (not relevant for comparison)
MongoDB / NoSQL (BASE)
 Basically Available: some response guaranteed
 Soft State: continuous (vs. discrete)
no transaction control, continuous processing
 Eventual Consistency: all-the-time? no
NoSQL pitfalls:
E-shop, Online banking, Stocks trading, … ($$$)
 Reason is that soft state is not plausible here.
 Transactional processing is a must (sending money from one account to another)
Business Intelligence, Reporting, Analysis, DWH, DM
 Reasons are:
o performance
 SQL: 20 years of complex queries optimizations
o functionality
 SQL: 20 years of out-of-the-box functionality
 CURSOR
 TRIGGER
 OVER
 ...
Lukáš Korous
lukas.korous@softica.eu
SQL <-> MongoDB
Structures
SQL MongoDB
Table Collection
Row Document
Column Field
Index Index
Join
Embedded Document
Linked Document
Select query
SQL MongoDB
SELECT * FROM
users
WHERE
age > 25 AND
age <= 50
ORDER BY
height
db.users
.find({
age:{ $gt: 25, $lte: 50 }
})
.sort({
height: 1
})
Simple update (invariant to current values)
SQL MongoDB
UPDATE
users
SET
db.users
.update (
{ age:{ $gt: 40, $lte: 50 } },
Lukáš Korous
lukas.korous@softica.eu
experience = ‘master’
WHERE
age > 40 AND
age <= 50
{ $set: { experience:‘master’} },
{ multi: true }
)
Update using old values
SQL MongoDB
UPDATE
users
SET
level = level + 1
WHERE
age > 40 AND
age <= 50
db.users
.update (
{ age:{ $gt: 40, $lte: 50 } },
{$inc: { level : 1} },
{ multi: true }
)
( *$inc: operator )
Authentication in MongoDB
Authenticate a User
To authenticate a user, either
 Use the command line authentication options (e.g. -u, -p, --authenticationDatabase) when
connecting to the mongod or mongos instance, or
 Connect first to the mongod or mongos instance, and then run the authenticate command
or the db.auth() method against the authentication database.
To authenticate, the client must authenticate the user against the user’s authentication
database.
For instance, if using the mongo shell as a client, you can specify the authentication
database for the user with the --authenticationDatabase option.
Centralized User Data
MongoDB stores all user information, including name, password, and the user's
authentication database, in the system.users collection in the admin database.
Lukáš Korous
lukas.korous@softica.eu
Enable Access Control
MongoDB does not enable access control by default. You can enable authorization using
the --auth or the security.authorization setting. Enabling internal authentication also
enables client authorization.
Once access control is enabled, users must authenticate themselves.
Roles
A role grants privileges to perform the specified actions on resource. Each privilege is
either specified explicitly in the role or inherited from another role or both.
Privileges
A privilege consists of a specified resource and the actions permitted on the resource.
A resource is either a database, collection, set of collections, or the cluster. If the resource is the
cluster, the affiliated actions affect the state of the system rather than a specific database or
collection.
An action specifies the operation allowed on the resource. It is an equivalent to SQL (MS-SQL,
MySQL, Oracle, ...) privileges.
Inherited Privileges
A role can include one or more existing roles in its definition, in which case the role inherits all
the privileges of the included roles.
A role can inherit privileges from other roles in its database. A role created on the admin
database can inherit privileges from roles in any database.
View Role’s Privileges
You can view the privileges for a role by issuing the rolesInfo command with the showPrivileges
and showBuiltinRoles fields both set to true.
Users and Roles
You can assign roles to users during the user creation. You can also update existing users to grant
or revoke roles.
A user assigned a role receives all the privileges of that role. A user can have multiple roles. By
assigning to the user roles in various databases, a user created in one database can have
permissions to act on other databases.
The first user created in the database should be a user administrator who has the privileges to
manage other users.
Lukáš Korous
lukas.korous@softica.eu
High Availability (HA) using MongoDB
Replica Set introduction
In this section, we are describing a 3-node replica set.
By using 3 nodes we have a 1-node failover cluster, generally using (2n + 1) nodes, we achieve n-
fail tolerant scheme.
A 3-node Replica Set has at any given point in time one Primary and two Secondary nodes.
Our client application / driver reads and writes from and to the cluster, and by design all writes
always go to the Primary node.
With Replica Sets, however, we can benefit from scaling of read operations, since these can be
routed to Secondary nodes:
But the main point of using Replica Sets is the HA / failover.
Replica Set – Failure of a Secondary node
Starts with a Secondary node not answering heartbeat messages and our client application /
driver stops getting responses to requests routed to the node.
Lukáš Korous
lukas.korous@softica.eu
Recovery process
The Primary node records that the Secondary node is inaccessible, therefore it stops trying to
replicate the write operations performed on the Primary node.
The client driver has to be configured in the way that it re-sends the read requests to other
nodes in the cluster (most drivers support this out-of-the-box). And thus the client gets a
response from one of the remaining nodes – with negligible downtime.
Lukáš Korous
lukas.korous@softica.eu
Replica Set – failure of a Primary node
Starts with a Secondary node not answering heartbeat messages and our client application /
driver stops getting responses to requests (both read and write) routed to the node. All timeout
intervals etc. are configurable on the MongoDB side.
This failure is a little bit more complicated than the failure of a Secondary node, since a new
Primary node has to be chosen (to be designated as the target of all write operations).
The process of selection is actually a voting mechanism, with a quorum-based voting algorithm in
use.
Quorum-based voting for replica control [source: en.wikipedia.org]
In replicated databases, a data object has copies present at several sites. To ensure
serializability, no two transactions should be allowed to read or write a data item
concurrently. In case of replicated databases, a quorum-based replica control protocol can
be used to ensure that no two copies of a data item are read or written by two transactions
concurrently.
The quorum-based voting for replica control is due to [Gifford, 1979].[3]
Each copy of a
replicated data item is assigned a vote. Each operation then has to obtain a read quorum
(Vr) or a write quorum (Vw) to read or write a data item, respectively. If a given data item
has a total of V votes, the quorums have to obey the following rules:
1. Vr + Vw > V
2. Vw > V/2
The first rule ensures that a data item is not read and written by two transactions
concurrently. Additionally, it ensures that a read quorum contains at least one site with the
Lukáš Korous
lukas.korous@softica.eu
newest version of the data item. The second rule ensures that two write operations from
two transactions cannot occur concurrently on the same data item. The two rules ensure
that one-copy serializability is maintained.
Oplog replication lag
It takes time to replicate the written data among nodes of the Replica set. This time can be
thought of as of a ‘lag’. This lag is not as important in standard operations mode – the eventual
consistency is an accepted and accounted for attribute of MongoDB database. Its importance
comes when the primary node fails and we have lost (non-recoverably) all data that have been
written to it in the time period before the crash of the same magnitude as the replication lag.
If we were to use MongoDB for critical data, we can adjust the write concern to return only when
data has reached a certain number of Secondary nodes (e.g. just one – as described in the
following illustration).
Lukáš Korous
lukas.korous@softica.eu
Replica Set – conclusion
 2n+1 servers => n-server fault tolerance
 Very fast recovery
o network-dependent (of course)
o on a common 1Gbit/s L2 network, with reasonable replication traffic at most ~ 100ms
 For monitoring it is important to query directly MongoDB cluster, not e.g. mongod services and
network connectivity (services may run, connectivity OK, but still the Replica Set may consider a
node down)
 Authentication for MongoDB is stored directly in the DB (automatic replication)
Lukáš Korous
lukas.korous@softica.eu
Use case #1: Stand-alone (non-replicated) MongoDB database backend
Installation
This installation process is for the Debian OS. Instructions for other Oss are available at
mongodb.org.
Get the necessary packages
sudo apt-get install -y mongodb-org
Log and data files location setup
The MongoDB instance stores its data files in /var/lib/mongodb and its log files in
/var/log/mongodb by default, and runs using the mongodb user account. You can specify
alternate log and data file directories in /etc/mongod.conf. See systemLog.path and
storage.dbPath for additional information:
sudo vim /etc/mongod.conf
...
# Where to store the data.
dbpath=/var/lib/mongodb
#where to log
logpath=/var/log/mongodb/mongodb.log
...
Starting the service
sudo service mongod start
Verify that the mongod process has started successfully by checking the contents of the log file at
/var/log/mongodb/mongod.log for a line reading:
[initandlisten] waiting for connections on port <port>
Lukáš Korous
lukas.korous@softica.eu
DB Setup
On the same system where the MongoDB is running, open a terminal window (or a command
prompt for Windows) and run the mongo shell with the following command:
mongo
In the mongo shell connected to a running mongod instance, switch to the test database (test
database is always present after installation):
use test
You can list all databases by issuing the following command:
> show dbs
local 0.78125GB
test 0.23012GB
And for creating a new DB, you can use:
use myNewDb
DB Operations
The same command for creating a database is used for switching among existing ones:
use myOldDb
Once the database is selected, the currently selected is queryable by issuing:
> db
myOldDb
For any query (update, select, insert), this variable – db – is used.
Insert query – “insert”
The following query inserts into the collection (~ table in RDBMS) ‘myProducts’ a new record
(~row) with the obvious attributes. An important notion is that we may issue this command
before creating the collection itself. The collection will be created for us in this case.
db.myProducts.insert({
productName : ‘Product ABC’,
productValue : 300,
Lukáš Korous
lukas.korous@softica.eu
productValueCurrency: ‘EUR’
})
Select query – “find”
Simple find is alternative to select * from TABLE (without any filter using the WHERE keyword in
SQL).
db.myProducts.find()
For the alternative to WHERE clause, the following is used (in this case, we are looking for
products with value of more than 200:
db.myProducts.find({productValue: {$qt: 200}})
Update query – “update”
Update query follows a similar pattern in SQL – specifying the filter in the WHERE clause. The
same similarity applies here:
db.myProducts.update({productValue: {$qt: 200}}, {$inc:
{productValue: 10}})
The last query looks for all product with value larger than 200 and increments the values of such
products by 10.
Use case #2: MongoDB Replica Set as a mobile app backend
Mobile app description
The app provides a complete solution for companies servicing lifts. It provides technicians with
the complete information about lift failures and required maintenances. It also enables the
technician to collect the information about his work, so the administrator receives predefined
protocols about lift reparation as well as the log book with working logs. Technicians can choose
and assign reported failures to them thus the application is a complete management solution for
service companies. The administrator can manage user accounts for his technicians and enable
or disable features.
Technologies used
End-user appliances: mobile phones, tablets, laptops
- each with a different client code (JavaScript), querying the same ReST API backend
o real-time communication facilitated by the Socket.IO library
Lukáš Korous
lukas.korous@softica.eu
ReST API offered by a Node.js server, connected to a MongoDB Replica Set using Mongoose.js
driver
Backend network overview
MongoDB config file
Relevant options only – this configuration file is used on all three nodes.
# mongod.conf
dbpath=/var/lib/mongodb
#where to log
logpath=/var/log/mongodb/mongod.log
logappend=true
#port = 27017
Server Purpose
smarlift Application backend
(Node.js)
zabbix Monitoring instance
running Zabbix server &
dashboard
mongo-
1
mongo-
2
mongo-
3
MongoDB Replica Set
Lukáš Korous
lukas.korous@softica.eu
# Listen to local interface only. Comment out to listen on all
interfaces.
#bind_ip = 127.0.0.1
# Turn on/off security. Off is currently the default
#noauth = true
auth = true
# Enable the HTTP interface (Defaults to port 28017).
httpinterface = true
# Replication Options
# in replicated mongo databases, specify the replica set name
here
replSet=initialReplicaSet
# path to a key file storing authentication info for
connections
# between replica set members
keyFile=/home/debian/mongodb-keyfile
Querying replica set state on a node
debian@mongo-1:~$ mongo -uroot –p*** --authenticationDatabase
admin
initialReplicaSet:SECONDARY> rs.config()
{
"_id" : "initialReplicaSet",
"version" : 9,
"members" : [
{
"_id" : 0,
"host" : "XXX.YYY.ZZZ.235:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
Lukáš Korous
lukas.korous@softica.eu
"priority" : 1,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
},
{
"_id" : 1,
"host" : "XXX.YYY.ZZZ.234:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
},
{
"_id" : 2,
"host" : "XXX.YYY.ZZZ.233:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatTimeoutSecs" : 10,
Lukáš Korous
lukas.korous@softica.eu
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
}
}
}
These settings can be changed:
initialReplicaSet:SECONDARY> new_config = {
settings.heartbeatTimeoutSecs : 3 }
initialReplicaSet:SECONDARY> rs.reconfig(new_config)
Lukáš Korous
lukas.korous@softica.eu
Schemas used
Here is a graphical representation of the main schemas used in the application. There are several
more schemas involved, but these form the main application logic.
Figure 1 - Basic schemas used
Schemas in the code
The syntax is Mongoose JSON. There are three entities here (the real application has more
entities).
For each, first the schema definition, and then an example of stored data are presented.
Green highlight is for linked documents
# User
var User = new Schema({
name: String,
email: String,
password: String,
Lukáš Korous
lukas.korous@softica.eu
reportsEmail: String,
car: String,
roles: [String],
pushToken: String,
created: Date
});
initialReplicaSet:SECONDARY> db.users.find()
initialReplicaSet:SECONDARY> ...,
{
"_id" : ObjectId("55f85cdf0b8869bd8f8c75ab"),
"car" : "6C97413",
"created" : ISODate("2015-09-23T21:00:00Z"),
"email" : "technik1@email.cz",
"name" : "Jakub Pospíšil",
"password" :
"$2a$10$kfnhy5OtbjM9SidZCvZJ0O0H2XkATFiEaP1za7lJS/0x7IvCDw.Ta",
"pushToken" :
"dRaltQD5QlI:APA91bHvEr71bG0jwfmNd3N5PxuWJfiMqMjRhgL_m_HxrMiIa_
x84S33uw0QRXGT1j4vzG-
nTiw_RPa8qyMs9z2rwFqakg3WKG6i0kmBgj0KFtALnsPasjpAqGCR86QeA9XLD_
Wq3rPq",
"roles" : [
"reports",
"maintenances",
"trips",
"elevators" ]
},
...
# Elevator
var Elevator = new Schema({
address: String,
label: String,
zone: Number,
maintenanceInterval: Number,
firstMaintenance: String,
contact: String,
Lukáš Korous
lukas.korous@softica.eu
contactPhone: [String],
contactEmail: [String],
lat: Number,
lng: Number,
creator: {
type: Schema.Types.ObjectId,
ref: "User"
},
created: Date,
lastModifiedBy: {
type: Schema.Types.ObjectId,
ref: "User"
},
lastModified: Date,
archived: Boolean
});
initialReplicaSet:SECONDARY> db.elevators.find()
initialReplicaSet:SECONDARY> ...,
{
"_id" : ObjectId("555555555555555555550154"),
"address" : "Novozámecká 284, Praha 9",
"maintenanceInterval" : 180,
"firstMaintenance" : "2015-03-17",
"contact" : "Král",
"contactPhone" : [
"602324770"
],
"contactEmail" : [
"Jakub.novaki@centrum.cz"
],
"zone" : 3,
"lat" : 50.0887121,
"lng" : 14.571461499999941,
"lastModified" : ISODate("2015-10-22T02:28:15.295Z"),
"lastModifiedBy" :
ObjectId("5579dd832d5d78ac2b68047c"),
Lukáš Korous
lukas.korous@softica.eu
"created" : ISODate("2015-10-22T02:28:15.295Z"),
"creator" : ObjectId("5579dd832d5d78ac2b68047c")
},
...
# Elevator maintenance
var Maintenance = new Schema({
elevator: {
type: Schema.Types.ObjectId,
ref: "Elevator"
},
technician: {
type: Schema.Types.ObjectId,
ref: "User"
},
datetimeFrom: Date,
datetimeTo: Date,
material: String,
note: String,
client: String,
progress: Number,
revisions: [{
rev: String,
done: Boolean,
_id: false
}],
created: Date,
lastModified: Date,
archived: Boolean
});
initialReplicaSet:SECONDARY> db.maintenances.find()
initialReplicaSet:SECONDARY> ...,
{
"__v" : 1,
Lukáš Korous
lukas.korous@softica.eu
"_id" : ObjectId("562de02a275008206945d60a"),
"client" : "ABB",
"created" : ISODate("2015-10-26T08:11:22.883Z"),
"datetimeFrom" : ISODate("2015-10-26T08:12:00Z"),
"datetimeTo" : ISODate("2015-10-26T08:27:00Z"),
"elevator" : ObjectId("555555555555555555550267"),
"lastModified" : ISODate("2015-10-26T08:12:23.391Z"),
"progress" : 2,
"revisions" : [
{
"rev" : "1",
"done" : true
},
{
"rev" : "2",
"done" : true
},
{
"rev" : "3",
"done" : true
}
],
"technician" : ObjectId("55f85cdf0b8869bd8f8c75ab")
}
ReST API on top of MongoDB
A typical set of API calls for an entity
var Schema = require("mongoose").Schema;
var _ = require("lodash");
var Elevator = new Schema({
address: String,
label: String,
Lukáš Korous
lukas.korous@softica.eu
zone: Number,
maintenanceInterval: Number,
firstMaintenance: String,
contact: String,
contactPhone: [String],
contactEmail: [String],
lat: Number,
lng: Number,
creator: {
type: Schema.Types.ObjectId,
ref: "User"
},
created: Date,
lastModifiedBy: {
type: Schema.Types.ObjectId,
ref: "User"
},
lastModified: Date,
archived: Boolean
});
# Functions
...
getAll: function() {
return this.find({archived: {$ne: true}});
},
# - translated to MongoDB shell as:
db.elevators.find({archived: {$ne: true}})
getById: function(id) {
return this.findById(id).then(function(elevator) {
if (elevator === null) {
throw new NotFoundError();
}
return elevator;
});
},
Lukáš Korous
lukas.korous@softica.eu
# - translated to MongoDB shell as:
db.elevators.find({_id: ObjectId(id)})
create: function(data, userId) {
var elevator = new this();
elevator.creator = userId;
elevator.created = new Date();
elevator.lastModifiedBy = userId;
elevator.lastModified = new Date();
elevator.assignData(data);
return elevator.save();
},
# - translated to MongoDB shell as:
db.elevators.insert({creator: ObjectId(userId), created:
ISODate(“2015-10-....”), lastModifiedBy: ObjectId(userId),
...})
update: function(id, data, userId) {
return this.findById(id).then(function(elevator) {
if (elevator === null) {
throw new NotFoundError();
}
elevator.lastModifiedBy = userId;
elevator.lastModified = new Date();
elevator.assignData(data);
return elevator.save();
});
},
# - translated to MongoDB shell as:
db.elevators.update({_id: ObjectId(id)}, { $set: {
lastModifiedBy: ObjectId(userId), ... } } )
archive: function(id, userId) {
return this.findById(id).then(function(elevator) {
if (elevator === null) {
throw new NotFoundError();
}
Lukáš Korous
lukas.korous@softica.eu
elevator.archived = true;
elevator.lastModifiedBy = userId;
elevator.lastModified = new Date();
return elevator.save();
}
# - translated to MongoDB shell as:
db.elevators.update({_id: ObjectId(id)}, { $set: {
archived: true, lastModifiedBy: ObjectId(userId), ... } } )
);
}
...
Lukáš Korous
lukas.korous@softica.eu
Monitoring
Zabbix (open source tool) for MongoDB monitoring
Zabbix can monitor the entire cluster (made of 3 nodes in our case) with an agent service running
on each of the nodes and an external server running the web GUI which is connected to the
agents.
Quantities monitorable with Zabbix
Items mongo-1 mongo-2 mongo-3
Agent ping Up (1) Up (1) Up (1)
Asserts: Total User Asserts 728 0 537
Asserts: Total Warning Asserts 0 0 0
Available memory 3.71 GB 3.64 GB 3.65 GB
Background Flushing: Background Flush Av
erage Time (ms)
5.61 94.5 5.57
Databases and Collections: List of All Datab
ase Stats
test:
collections=3,...
test:
collections=3,...
-
Databases and Collections: List of Databas
e Average Object Size
test=60.8 ||
showcas...
test=60.8 ||
showcas...
-
Databases and Collections: List of Databas
e Collection Count
test=3 ||
showcase=1...
test=3 ||
showcase=1...
-
Databases and Collections: List of Databas
e Data Size
test=304 ||
showcase...
test=304 ||
showcase...
-
Lukáš Korous
lukas.korous@softica.eu
Databases and Collections: List of Databas
e Extent Count
test=3 ||
showcase=2...
test=3 ||
showcase=2...
-
Databases and Collections: List of Databas
e File Size
test=67108864
|| sho...
test=67108864
|| sho...
-
Databases and Collections: List of Databas
e Index Count
test=1 ||
showcase=1...
test=1 ||
showcase=1...
-
Databases and Collections: List of Databas
e Index Size
test=8176 ||
showcas...
test=8176 ||
showcas...
-
Databases and Collections: List of Databas
e Object Count
test=5 ||
showcase=2...
test=5 ||
showcase=2...
-
Databases and Collections: List of Databas
e Storage Size
test=20480 ||
showca...
test=20480 ||
showca...
-
Free disk space on / 49.16 GB 50.2 GB 53.2 GB
Free disk space on / (percentage) 81.41 % 83.14 % 88.11 %
Host boot time
2015-09-26
17:14:27
2015-09-18
16:59:31
2015-09-18
16:59:36
Host local time
2015-11-23
17:16:37
2015-11-23
17:16:13
2015-11-23
17:16:54
Host name mongo-1 mongo-2 mongo-3
Host name of zabbix_agentd running mongo-1 mongo-2 mongo-3
Maximum number of opened files 404256 404257 404257
Maximum number of processes 32768 32768 32768
Memory: Heap Memory Size (MB) 510.74 500.73 500.86
Memory: Memory Addressing (32/64 bit) 64 64 64
Memory: Page Faults/minute 0 0 0
Memory: Resident Memory Size (MB) 679 681 682
Memory: Virtual Memory Size (MB) 11193 11179 11181
Miscellaneous: Data Collector 0 0 0
Miscellaneous: MongoDB Plugin Checksu
m
db987f88e0a81
2a16f59...
db987f88e0a812
a16f59...
db987f88e0a812
a16f59...
Miscellaneous: MongoDB Plugin Data Colle
ction Time (seconds)
0 0 0
Miscellaneous: MongoDB Plugin Version 0.4 0.4 0.4
Lukáš Korous
lukas.korous@softica.eu
Miscellaneous: MongoDB Uptime (seconds
)
5014923 1011 5009683
Miscellaneous: MongoDB Version 3.0.6 3.0.6 3.0.6
Number of processes 89 89 89
Replication: Count of ReplicaSet Members
Needing Attention
0 0 0
Replication: Entries in oplog.rs Collection 1 2792 2792
Replication: Is Mongo Server Part of a Repl
icatSet
Yes Yes Yes
Replication: List of ReplicaSet Members in
Attention State
empty empty empty
Replication: ReplicaSet Healthy Member C
ount
3 3 3
Replication: ReplicaSet Host Names
host#0 =
XXX.YYY.ZZZ....
host#0 =
XXX.YYY.ZZZ....
host#0 =
XXX.YYY.ZZZ....
Replication: ReplicaSet Member Count 3 3 3
Replication: ReplicaSet Name initialReplicaSet initialReplicaSet initialReplicaSet
Sharding: Is Mongo Server a Cluster Router
(mongos process)
No No No
System information
Linux mongo-1
3.16.0...
Linux mongo-2
3.16.0...
Linux mongo-3
3.16.0...
System uptime
58 days,
00:53:16
66 days,
01:10:48
66 days,
01:11:24
Total disk space on / 62.96 GB 62.96 GB 62.96 GB
Total memory 3.87 GB 3.87 GB 3.87 GB
Total swap space 0 B 0 B 0 B
Used disk space on / 11.22 GB 10.18 GB 7.18 GB
Version of zabbix_agent(d) running 2.2.7 2.2.7 2.2.7
Services running on the host
root@mongo-3:/home/debian# service --status-all
...
[ + ] mongod (mongoDB service)
...
Lukáš Korous
lukas.korous@softica.eu
[ + ] zabbix-agent (monitoring agent service)
...
root@mongo-3:/home/debian#
Lukáš Korous
lukas.korous@softica.eu
Ports on which MongoDB and Zabbix listen
root@mongo-3:/home/debian# netstat -ap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
...
tcp 0 0 *:zabbix-agent *:* LISTEN
8982/zabbix_agentd
tcp 0 0 *:27017 *:* LISTEN
9294/mongod
...
MongoDB logfile - /var/log/mongodb/mongodb.log
...
2015-11-30T09:31:23.920+0000 I ACCESS [conn465825]
Successfully authenticated as principal __system on local
2015-11-30T09:31:39.905+0000 I NETWORK [conn465824] end
connection XXX.YYY.ZZZ.235:33007 (1 connection now open)
2015-11-30T09:31:39.906+0000 I NETWORK [initandlisten]
connection accepted from XXX.YYY.ZZZ.235:33009 #465826 (3
connections now open)
2015-11-30T09:31:40.023+0000 I ACCESS [conn465826]
Successfully authenticated as principal __system on local
2015-11-30T09:31:53.942+0000 I NETWORK [conn465825] end
connection XXX.YYY.ZZZ.234:60287 (1 connection now open)
2015-11-30T09:31:53.943+0000 I NETWORK [initandlisten]
connection accepted from XXX.YYY.ZZZ.234:60289 #465827 (2
connections now open)
2015-11-30T09:31:53.965+0000 I ACCESS [conn465827]
Successfully authenticated as principal __system on local
2015-11-30T09:31:56.422+0000 I NETWORK [initandlisten]
connection accepted from 10.20.0.6:36444 #465828 (3 connections
now open)
2015-11-30T09:31:56.448+0000 I ACCESS [conn465828]
Successfully authenticated as principal root on admin
2015-11-30T09:31:56.457+0000 I NETWORK [conn465828] end
connection 10.20.0.6:36444 (2 connections now open)
...
Lukáš Korous
lukas.korous@softica.eu
Sources
 wikipedia.org
 mongodb.org
 Softica Solutions private repositories

More Related Content

PDF
Another example PDF
PDF
Jscriptdeviationsfromes3
PDF
Parallels Plesk Panel 9 Client's Guide
PDF
Rails Cookbook
PDF
The Total Book Developing Solutions With EPiServer 4
PDF
html-css-bootstrap-javascript-and-jquery
PDF
ใบงาน 1-เทคโนโลยีสารสนเทศ
PDF
Ug recording excelmacros
Another example PDF
Jscriptdeviationsfromes3
Parallels Plesk Panel 9 Client's Guide
Rails Cookbook
The Total Book Developing Solutions With EPiServer 4
html-css-bootstrap-javascript-and-jquery
ใบงาน 1-เทคโนโลยีสารสนเทศ
Ug recording excelmacros

What's hot (18)

PDF
The MySQL Cluster API Developer Guide
PDF
PlayBook userguide
PDF
Datavideo CG-500 SD/HD Advanced Timeline based CG for live/post production
PDF
Hp networking-and-cisco-cli-reference-guide june-10_ww_eng_ltr
PDF
C sharp programming
PDF
Excel vba notesforprofessionals
PDF
Python programming
PDF
PHP-Nuke-HOWTO
PDF
Configuration vm sur hyper
PDF
Mvc music store tutorial - v3.0
PDF
Linux training
PDF
Ppm7.5 cmd tokval
PDF
PDF
Hp man ppm9.20_whats_new_pdf
PDF
Begining j2 me
PDF
Ppm7.5 demand cg
PDF
Spec
PDF
TopStyle Help &amp; &lt;b>Tutorial&lt;/b>
The MySQL Cluster API Developer Guide
PlayBook userguide
Datavideo CG-500 SD/HD Advanced Timeline based CG for live/post production
Hp networking-and-cisco-cli-reference-guide june-10_ww_eng_ltr
C sharp programming
Excel vba notesforprofessionals
Python programming
PHP-Nuke-HOWTO
Configuration vm sur hyper
Mvc music store tutorial - v3.0
Linux training
Ppm7.5 cmd tokval
Hp man ppm9.20_whats_new_pdf
Begining j2 me
Ppm7.5 demand cg
Spec
TopStyle Help &amp; &lt;b>Tutorial&lt;/b>
Ad

Similar to MongoDB Use Case - Mobile App Backend (20)

PDF
Tortoise svn 1.7-en
PDF
C sharp programming[1]
PDF
Java script tools guide cs6
PDF
R Data
PDF
Another example PDF
PDF
Junipe 1
PDF
Blackberry v.6.0
PDF
R installation and administration
PDF
advanced java.pdf
PDF
Flash File Format Specification
PDF
Ebay News 1998 10 27 Earnings
PDF
W java81
PDF
Photoshop cs4 scripting guide
PDF
Photoshop cs4 scripting_guide
PDF
Documentation de Doctrine ORM
PDF
Docker containerization cookbook
PDF
Mx Odbc
PDF
Mvc music store tutorial - v3.0 (1)
PDF
Mvc music store tutorial - v3.0
PDF
Mvc music store tutorial - v3.0
Tortoise svn 1.7-en
C sharp programming[1]
Java script tools guide cs6
R Data
Another example PDF
Junipe 1
Blackberry v.6.0
R installation and administration
advanced java.pdf
Flash File Format Specification
Ebay News 1998 10 27 Earnings
W java81
Photoshop cs4 scripting guide
Photoshop cs4 scripting_guide
Documentation de Doctrine ORM
Docker containerization cookbook
Mx Odbc
Mvc music store tutorial - v3.0 (1)
Mvc music store tutorial - v3.0
Mvc music store tutorial - v3.0
Ad

Recently uploaded (20)

PDF
Navsoft: AI-Powered Business Solutions & Custom Software Development
PPTX
ai tools demonstartion for schools and inter college
PPTX
Computer Software and OS of computer science of grade 11.pptx
PPTX
CHAPTER 2 - PM Management and IT Context
PDF
Upgrade and Innovation Strategies for SAP ERP Customers
PDF
Design an Analysis of Algorithms II-SECS-1021-03
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
PPT
Introduction Database Management System for Course Database
PPTX
history of c programming in notes for students .pptx
PPTX
Reimagine Home Health with the Power of Agentic AI​
PDF
Odoo Companies in India – Driving Business Transformation.pdf
PPTX
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
PDF
medical staffing services at VALiNTRY
PDF
PTS Company Brochure 2025 (1).pdf.......
PPTX
Operating system designcfffgfgggggggvggggggggg
PDF
Designing Intelligence for the Shop Floor.pdf
Navsoft: AI-Powered Business Solutions & Custom Software Development
ai tools demonstartion for schools and inter college
Computer Software and OS of computer science of grade 11.pptx
CHAPTER 2 - PM Management and IT Context
Upgrade and Innovation Strategies for SAP ERP Customers
Design an Analysis of Algorithms II-SECS-1021-03
VVF-Customer-Presentation2025-Ver1.9.pptx
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
Internet Downloader Manager (IDM) Crack 6.42 Build 41
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
Introduction Database Management System for Course Database
history of c programming in notes for students .pptx
Reimagine Home Health with the Power of Agentic AI​
Odoo Companies in India – Driving Business Transformation.pdf
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
medical staffing services at VALiNTRY
PTS Company Brochure 2025 (1).pdf.......
Operating system designcfffgfgggggggvggggggggg
Designing Intelligence for the Shop Floor.pdf

MongoDB Use Case - Mobile App Backend

  • 1. Lukáš Korous lukas.korous@softica.eu Document database MongoDB introduction & mobile app backend use case Lukáš Korous
  • 2. Lukáš Korous lukas.korous@softica.eu Contents Introduction ......................................................................................................................................5 Characteristics...........................................................................................................................................5 NoSQL pitfalls:...........................................................................................................................................6 E-shop, Online banking, Stocks trading, … ($$$) ..................................................................................6 Business Intelligence, Reporting, Analysis, DWH, DM..........................................................................6 SQL <-> MongoDB.....................................................................................................................................7 Structures..............................................................................................................................................7 SQL ........................................................................................................................................................7 MongoDB ..............................................................................................................................................7 Select query ..........................................................................................................................................7 SQL ........................................................................................................................................................7 MongoDB ..............................................................................................................................................7 Simple update (invariant to current values).........................................................................................7 SQL ........................................................................................................................................................7 MongoDB ..............................................................................................................................................7 Update using old values........................................................................................................................8 SQL ........................................................................................................................................................8 MongoDB ..............................................................................................................................................8 Authentication in MongoDB .............................................................................................................8 Authenticate a User ..................................................................................................................................8 Centralized User Data ...............................................................................................................................8 Enable Access Control...............................................................................................................................9 Roles..........................................................................................................................................................9 Privileges...................................................................................................................................................9 Inherited Privileges ...............................................................................................................................9 View Role’s Privileges............................................................................................................................9 Users and Roles.........................................................................................................................................9 High Availability (HA) using MongoDB............................................................................................10 Replica Set introduction..........................................................................................................................10 Replica Set – Failure of a Secondary node..............................................................................................10
  • 3. Lukáš Korous lukas.korous@softica.eu Recovery process ................................................................................................................................11 Replica Set – failure of a Primary node...................................................................................................12 Quorum-based voting for replica control [source: en.wikipedia.org]................................................12 Oplog replication lag...............................................................................................................................13 Replica Set – conclusion..........................................................................................................................14 Use case #1: Stand-alone (non-replicated) MongoDB database backend......................................15 Installation ..............................................................................................................................................15 Get the necessary packages................................................................................................................15 Log and data files location setup ........................................................................................................15 Starting the service .............................................................................................................................15 DB Setup..................................................................................................................................................16 DB Operations.........................................................................................................................................16 Insert query – “insert” ........................................................................................................................16 Select query – “find”...........................................................................................................................17 Update query – “update”....................................................................................................................17 Use case #2: MongoDB Replica Set as a mobile app backend........................................................17 Mobile app description...........................................................................................................................17 Technologies used...................................................................................................................................17 Backend network overview ....................................................................................................................18 MongoDB config file................................................................................................................................18 Querying replica set state on a node......................................................................................................19 Schemas used..........................................................................................................................................22 Schemas in the code ...........................................................................................................................22 ReST API on top of MongoDB .................................................................................................................26 Monitoring ..............................................................................................................................................30 Zabbix (open source tool) for MongoDB monitoring..........................................................................30 Quantities monitorable with Zabbix ...................................................................................................30 Services running on the host ..............................................................................................................32 Ports on which MongoDB and Zabbix listen.......................................................................................34 MongoDB logfile - /var/log/mongodb/mongodb.log .........................................................................34 Sources............................................................................................................................................35
  • 5. Lukáš Korous lukas.korous@softica.eu Introduction Characteristics MongoDB is a cross-platform document-oriented database. Storage pattern is a collection of JSON-like documents with dynamic schemas (MongoDB calls the format BSON – binary JSON – and it is a binary format), making the integration of data in certain types of applications easier and faster. Here is an example of a record: Released under a combination of the GNU Affero General Public License and the Apache License, MongoDB is free and open-source software. MongoDB is the most popular NoSQL database in the world:
  • 6. Lukáš Korous lukas.korous@softica.eu SQL vs. NoSQL SQL (ACID)  Atomicity: all-or-nothing (transactional processing)  Consistency: all-the-time? yes  Isolation: discrete (vs. continuous)  Durability (not relevant for comparison) MongoDB / NoSQL (BASE)  Basically Available: some response guaranteed  Soft State: continuous (vs. discrete) no transaction control, continuous processing  Eventual Consistency: all-the-time? no NoSQL pitfalls: E-shop, Online banking, Stocks trading, … ($$$)  Reason is that soft state is not plausible here.  Transactional processing is a must (sending money from one account to another) Business Intelligence, Reporting, Analysis, DWH, DM  Reasons are: o performance  SQL: 20 years of complex queries optimizations o functionality  SQL: 20 years of out-of-the-box functionality  CURSOR  TRIGGER  OVER  ...
  • 7. Lukáš Korous lukas.korous@softica.eu SQL <-> MongoDB Structures SQL MongoDB Table Collection Row Document Column Field Index Index Join Embedded Document Linked Document Select query SQL MongoDB SELECT * FROM users WHERE age > 25 AND age <= 50 ORDER BY height db.users .find({ age:{ $gt: 25, $lte: 50 } }) .sort({ height: 1 }) Simple update (invariant to current values) SQL MongoDB UPDATE users SET db.users .update ( { age:{ $gt: 40, $lte: 50 } },
  • 8. Lukáš Korous lukas.korous@softica.eu experience = ‘master’ WHERE age > 40 AND age <= 50 { $set: { experience:‘master’} }, { multi: true } ) Update using old values SQL MongoDB UPDATE users SET level = level + 1 WHERE age > 40 AND age <= 50 db.users .update ( { age:{ $gt: 40, $lte: 50 } }, {$inc: { level : 1} }, { multi: true } ) ( *$inc: operator ) Authentication in MongoDB Authenticate a User To authenticate a user, either  Use the command line authentication options (e.g. -u, -p, --authenticationDatabase) when connecting to the mongod or mongos instance, or  Connect first to the mongod or mongos instance, and then run the authenticate command or the db.auth() method against the authentication database. To authenticate, the client must authenticate the user against the user’s authentication database. For instance, if using the mongo shell as a client, you can specify the authentication database for the user with the --authenticationDatabase option. Centralized User Data MongoDB stores all user information, including name, password, and the user's authentication database, in the system.users collection in the admin database.
  • 9. Lukáš Korous lukas.korous@softica.eu Enable Access Control MongoDB does not enable access control by default. You can enable authorization using the --auth or the security.authorization setting. Enabling internal authentication also enables client authorization. Once access control is enabled, users must authenticate themselves. Roles A role grants privileges to perform the specified actions on resource. Each privilege is either specified explicitly in the role or inherited from another role or both. Privileges A privilege consists of a specified resource and the actions permitted on the resource. A resource is either a database, collection, set of collections, or the cluster. If the resource is the cluster, the affiliated actions affect the state of the system rather than a specific database or collection. An action specifies the operation allowed on the resource. It is an equivalent to SQL (MS-SQL, MySQL, Oracle, ...) privileges. Inherited Privileges A role can include one or more existing roles in its definition, in which case the role inherits all the privileges of the included roles. A role can inherit privileges from other roles in its database. A role created on the admin database can inherit privileges from roles in any database. View Role’s Privileges You can view the privileges for a role by issuing the rolesInfo command with the showPrivileges and showBuiltinRoles fields both set to true. Users and Roles You can assign roles to users during the user creation. You can also update existing users to grant or revoke roles. A user assigned a role receives all the privileges of that role. A user can have multiple roles. By assigning to the user roles in various databases, a user created in one database can have permissions to act on other databases. The first user created in the database should be a user administrator who has the privileges to manage other users.
  • 10. Lukáš Korous lukas.korous@softica.eu High Availability (HA) using MongoDB Replica Set introduction In this section, we are describing a 3-node replica set. By using 3 nodes we have a 1-node failover cluster, generally using (2n + 1) nodes, we achieve n- fail tolerant scheme. A 3-node Replica Set has at any given point in time one Primary and two Secondary nodes. Our client application / driver reads and writes from and to the cluster, and by design all writes always go to the Primary node. With Replica Sets, however, we can benefit from scaling of read operations, since these can be routed to Secondary nodes: But the main point of using Replica Sets is the HA / failover. Replica Set – Failure of a Secondary node Starts with a Secondary node not answering heartbeat messages and our client application / driver stops getting responses to requests routed to the node.
  • 11. Lukáš Korous lukas.korous@softica.eu Recovery process The Primary node records that the Secondary node is inaccessible, therefore it stops trying to replicate the write operations performed on the Primary node. The client driver has to be configured in the way that it re-sends the read requests to other nodes in the cluster (most drivers support this out-of-the-box). And thus the client gets a response from one of the remaining nodes – with negligible downtime.
  • 12. Lukáš Korous lukas.korous@softica.eu Replica Set – failure of a Primary node Starts with a Secondary node not answering heartbeat messages and our client application / driver stops getting responses to requests (both read and write) routed to the node. All timeout intervals etc. are configurable on the MongoDB side. This failure is a little bit more complicated than the failure of a Secondary node, since a new Primary node has to be chosen (to be designated as the target of all write operations). The process of selection is actually a voting mechanism, with a quorum-based voting algorithm in use. Quorum-based voting for replica control [source: en.wikipedia.org] In replicated databases, a data object has copies present at several sites. To ensure serializability, no two transactions should be allowed to read or write a data item concurrently. In case of replicated databases, a quorum-based replica control protocol can be used to ensure that no two copies of a data item are read or written by two transactions concurrently. The quorum-based voting for replica control is due to [Gifford, 1979].[3] Each copy of a replicated data item is assigned a vote. Each operation then has to obtain a read quorum (Vr) or a write quorum (Vw) to read or write a data item, respectively. If a given data item has a total of V votes, the quorums have to obey the following rules: 1. Vr + Vw > V 2. Vw > V/2 The first rule ensures that a data item is not read and written by two transactions concurrently. Additionally, it ensures that a read quorum contains at least one site with the
  • 13. Lukáš Korous lukas.korous@softica.eu newest version of the data item. The second rule ensures that two write operations from two transactions cannot occur concurrently on the same data item. The two rules ensure that one-copy serializability is maintained. Oplog replication lag It takes time to replicate the written data among nodes of the Replica set. This time can be thought of as of a ‘lag’. This lag is not as important in standard operations mode – the eventual consistency is an accepted and accounted for attribute of MongoDB database. Its importance comes when the primary node fails and we have lost (non-recoverably) all data that have been written to it in the time period before the crash of the same magnitude as the replication lag. If we were to use MongoDB for critical data, we can adjust the write concern to return only when data has reached a certain number of Secondary nodes (e.g. just one – as described in the following illustration).
  • 14. Lukáš Korous lukas.korous@softica.eu Replica Set – conclusion  2n+1 servers => n-server fault tolerance  Very fast recovery o network-dependent (of course) o on a common 1Gbit/s L2 network, with reasonable replication traffic at most ~ 100ms  For monitoring it is important to query directly MongoDB cluster, not e.g. mongod services and network connectivity (services may run, connectivity OK, but still the Replica Set may consider a node down)  Authentication for MongoDB is stored directly in the DB (automatic replication)
  • 15. Lukáš Korous lukas.korous@softica.eu Use case #1: Stand-alone (non-replicated) MongoDB database backend Installation This installation process is for the Debian OS. Instructions for other Oss are available at mongodb.org. Get the necessary packages sudo apt-get install -y mongodb-org Log and data files location setup The MongoDB instance stores its data files in /var/lib/mongodb and its log files in /var/log/mongodb by default, and runs using the mongodb user account. You can specify alternate log and data file directories in /etc/mongod.conf. See systemLog.path and storage.dbPath for additional information: sudo vim /etc/mongod.conf ... # Where to store the data. dbpath=/var/lib/mongodb #where to log logpath=/var/log/mongodb/mongodb.log ... Starting the service sudo service mongod start Verify that the mongod process has started successfully by checking the contents of the log file at /var/log/mongodb/mongod.log for a line reading: [initandlisten] waiting for connections on port <port>
  • 16. Lukáš Korous lukas.korous@softica.eu DB Setup On the same system where the MongoDB is running, open a terminal window (or a command prompt for Windows) and run the mongo shell with the following command: mongo In the mongo shell connected to a running mongod instance, switch to the test database (test database is always present after installation): use test You can list all databases by issuing the following command: > show dbs local 0.78125GB test 0.23012GB And for creating a new DB, you can use: use myNewDb DB Operations The same command for creating a database is used for switching among existing ones: use myOldDb Once the database is selected, the currently selected is queryable by issuing: > db myOldDb For any query (update, select, insert), this variable – db – is used. Insert query – “insert” The following query inserts into the collection (~ table in RDBMS) ‘myProducts’ a new record (~row) with the obvious attributes. An important notion is that we may issue this command before creating the collection itself. The collection will be created for us in this case. db.myProducts.insert({ productName : ‘Product ABC’, productValue : 300,
  • 17. Lukáš Korous lukas.korous@softica.eu productValueCurrency: ‘EUR’ }) Select query – “find” Simple find is alternative to select * from TABLE (without any filter using the WHERE keyword in SQL). db.myProducts.find() For the alternative to WHERE clause, the following is used (in this case, we are looking for products with value of more than 200: db.myProducts.find({productValue: {$qt: 200}}) Update query – “update” Update query follows a similar pattern in SQL – specifying the filter in the WHERE clause. The same similarity applies here: db.myProducts.update({productValue: {$qt: 200}}, {$inc: {productValue: 10}}) The last query looks for all product with value larger than 200 and increments the values of such products by 10. Use case #2: MongoDB Replica Set as a mobile app backend Mobile app description The app provides a complete solution for companies servicing lifts. It provides technicians with the complete information about lift failures and required maintenances. It also enables the technician to collect the information about his work, so the administrator receives predefined protocols about lift reparation as well as the log book with working logs. Technicians can choose and assign reported failures to them thus the application is a complete management solution for service companies. The administrator can manage user accounts for his technicians and enable or disable features. Technologies used End-user appliances: mobile phones, tablets, laptops - each with a different client code (JavaScript), querying the same ReST API backend o real-time communication facilitated by the Socket.IO library
  • 18. Lukáš Korous lukas.korous@softica.eu ReST API offered by a Node.js server, connected to a MongoDB Replica Set using Mongoose.js driver Backend network overview MongoDB config file Relevant options only – this configuration file is used on all three nodes. # mongod.conf dbpath=/var/lib/mongodb #where to log logpath=/var/log/mongodb/mongod.log logappend=true #port = 27017 Server Purpose smarlift Application backend (Node.js) zabbix Monitoring instance running Zabbix server & dashboard mongo- 1 mongo- 2 mongo- 3 MongoDB Replica Set
  • 19. Lukáš Korous lukas.korous@softica.eu # Listen to local interface only. Comment out to listen on all interfaces. #bind_ip = 127.0.0.1 # Turn on/off security. Off is currently the default #noauth = true auth = true # Enable the HTTP interface (Defaults to port 28017). httpinterface = true # Replication Options # in replicated mongo databases, specify the replica set name here replSet=initialReplicaSet # path to a key file storing authentication info for connections # between replica set members keyFile=/home/debian/mongodb-keyfile Querying replica set state on a node debian@mongo-1:~$ mongo -uroot –p*** --authenticationDatabase admin initialReplicaSet:SECONDARY> rs.config() { "_id" : "initialReplicaSet", "version" : 9, "members" : [ { "_id" : 0, "host" : "XXX.YYY.ZZZ.235:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false,
  • 20. Lukáš Korous lukas.korous@softica.eu "priority" : 1, "tags" : { }, "slaveDelay" : 0, "votes" : 1 }, { "_id" : 1, "host" : "XXX.YYY.ZZZ.234:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : 0, "votes" : 1 }, { "_id" : 2, "host" : "XXX.YYY.ZZZ.233:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : 0, "votes" : 1 } ], "settings" : { "chainingAllowed" : true, "heartbeatTimeoutSecs" : 10,
  • 21. Lukáš Korous lukas.korous@softica.eu "getLastErrorModes" : { }, "getLastErrorDefaults" : { "w" : 1, "wtimeout" : 0 } } } These settings can be changed: initialReplicaSet:SECONDARY> new_config = { settings.heartbeatTimeoutSecs : 3 } initialReplicaSet:SECONDARY> rs.reconfig(new_config)
  • 22. Lukáš Korous lukas.korous@softica.eu Schemas used Here is a graphical representation of the main schemas used in the application. There are several more schemas involved, but these form the main application logic. Figure 1 - Basic schemas used Schemas in the code The syntax is Mongoose JSON. There are three entities here (the real application has more entities). For each, first the schema definition, and then an example of stored data are presented. Green highlight is for linked documents # User var User = new Schema({ name: String, email: String, password: String,
  • 23. Lukáš Korous lukas.korous@softica.eu reportsEmail: String, car: String, roles: [String], pushToken: String, created: Date }); initialReplicaSet:SECONDARY> db.users.find() initialReplicaSet:SECONDARY> ..., { "_id" : ObjectId("55f85cdf0b8869bd8f8c75ab"), "car" : "6C97413", "created" : ISODate("2015-09-23T21:00:00Z"), "email" : "technik1@email.cz", "name" : "Jakub Pospíšil", "password" : "$2a$10$kfnhy5OtbjM9SidZCvZJ0O0H2XkATFiEaP1za7lJS/0x7IvCDw.Ta", "pushToken" : "dRaltQD5QlI:APA91bHvEr71bG0jwfmNd3N5PxuWJfiMqMjRhgL_m_HxrMiIa_ x84S33uw0QRXGT1j4vzG- nTiw_RPa8qyMs9z2rwFqakg3WKG6i0kmBgj0KFtALnsPasjpAqGCR86QeA9XLD_ Wq3rPq", "roles" : [ "reports", "maintenances", "trips", "elevators" ] }, ... # Elevator var Elevator = new Schema({ address: String, label: String, zone: Number, maintenanceInterval: Number, firstMaintenance: String, contact: String,
  • 24. Lukáš Korous lukas.korous@softica.eu contactPhone: [String], contactEmail: [String], lat: Number, lng: Number, creator: { type: Schema.Types.ObjectId, ref: "User" }, created: Date, lastModifiedBy: { type: Schema.Types.ObjectId, ref: "User" }, lastModified: Date, archived: Boolean }); initialReplicaSet:SECONDARY> db.elevators.find() initialReplicaSet:SECONDARY> ..., { "_id" : ObjectId("555555555555555555550154"), "address" : "Novozámecká 284, Praha 9", "maintenanceInterval" : 180, "firstMaintenance" : "2015-03-17", "contact" : "Král", "contactPhone" : [ "602324770" ], "contactEmail" : [ "Jakub.novaki@centrum.cz" ], "zone" : 3, "lat" : 50.0887121, "lng" : 14.571461499999941, "lastModified" : ISODate("2015-10-22T02:28:15.295Z"), "lastModifiedBy" : ObjectId("5579dd832d5d78ac2b68047c"),
  • 25. Lukáš Korous lukas.korous@softica.eu "created" : ISODate("2015-10-22T02:28:15.295Z"), "creator" : ObjectId("5579dd832d5d78ac2b68047c") }, ... # Elevator maintenance var Maintenance = new Schema({ elevator: { type: Schema.Types.ObjectId, ref: "Elevator" }, technician: { type: Schema.Types.ObjectId, ref: "User" }, datetimeFrom: Date, datetimeTo: Date, material: String, note: String, client: String, progress: Number, revisions: [{ rev: String, done: Boolean, _id: false }], created: Date, lastModified: Date, archived: Boolean }); initialReplicaSet:SECONDARY> db.maintenances.find() initialReplicaSet:SECONDARY> ..., { "__v" : 1,
  • 26. Lukáš Korous lukas.korous@softica.eu "_id" : ObjectId("562de02a275008206945d60a"), "client" : "ABB", "created" : ISODate("2015-10-26T08:11:22.883Z"), "datetimeFrom" : ISODate("2015-10-26T08:12:00Z"), "datetimeTo" : ISODate("2015-10-26T08:27:00Z"), "elevator" : ObjectId("555555555555555555550267"), "lastModified" : ISODate("2015-10-26T08:12:23.391Z"), "progress" : 2, "revisions" : [ { "rev" : "1", "done" : true }, { "rev" : "2", "done" : true }, { "rev" : "3", "done" : true } ], "technician" : ObjectId("55f85cdf0b8869bd8f8c75ab") } ReST API on top of MongoDB A typical set of API calls for an entity var Schema = require("mongoose").Schema; var _ = require("lodash"); var Elevator = new Schema({ address: String, label: String,
  • 27. Lukáš Korous lukas.korous@softica.eu zone: Number, maintenanceInterval: Number, firstMaintenance: String, contact: String, contactPhone: [String], contactEmail: [String], lat: Number, lng: Number, creator: { type: Schema.Types.ObjectId, ref: "User" }, created: Date, lastModifiedBy: { type: Schema.Types.ObjectId, ref: "User" }, lastModified: Date, archived: Boolean }); # Functions ... getAll: function() { return this.find({archived: {$ne: true}}); }, # - translated to MongoDB shell as: db.elevators.find({archived: {$ne: true}}) getById: function(id) { return this.findById(id).then(function(elevator) { if (elevator === null) { throw new NotFoundError(); } return elevator; }); },
  • 28. Lukáš Korous lukas.korous@softica.eu # - translated to MongoDB shell as: db.elevators.find({_id: ObjectId(id)}) create: function(data, userId) { var elevator = new this(); elevator.creator = userId; elevator.created = new Date(); elevator.lastModifiedBy = userId; elevator.lastModified = new Date(); elevator.assignData(data); return elevator.save(); }, # - translated to MongoDB shell as: db.elevators.insert({creator: ObjectId(userId), created: ISODate(“2015-10-....”), lastModifiedBy: ObjectId(userId), ...}) update: function(id, data, userId) { return this.findById(id).then(function(elevator) { if (elevator === null) { throw new NotFoundError(); } elevator.lastModifiedBy = userId; elevator.lastModified = new Date(); elevator.assignData(data); return elevator.save(); }); }, # - translated to MongoDB shell as: db.elevators.update({_id: ObjectId(id)}, { $set: { lastModifiedBy: ObjectId(userId), ... } } ) archive: function(id, userId) { return this.findById(id).then(function(elevator) { if (elevator === null) { throw new NotFoundError(); }
  • 29. Lukáš Korous lukas.korous@softica.eu elevator.archived = true; elevator.lastModifiedBy = userId; elevator.lastModified = new Date(); return elevator.save(); } # - translated to MongoDB shell as: db.elevators.update({_id: ObjectId(id)}, { $set: { archived: true, lastModifiedBy: ObjectId(userId), ... } } ) ); } ...
  • 30. Lukáš Korous lukas.korous@softica.eu Monitoring Zabbix (open source tool) for MongoDB monitoring Zabbix can monitor the entire cluster (made of 3 nodes in our case) with an agent service running on each of the nodes and an external server running the web GUI which is connected to the agents. Quantities monitorable with Zabbix Items mongo-1 mongo-2 mongo-3 Agent ping Up (1) Up (1) Up (1) Asserts: Total User Asserts 728 0 537 Asserts: Total Warning Asserts 0 0 0 Available memory 3.71 GB 3.64 GB 3.65 GB Background Flushing: Background Flush Av erage Time (ms) 5.61 94.5 5.57 Databases and Collections: List of All Datab ase Stats test: collections=3,... test: collections=3,... - Databases and Collections: List of Databas e Average Object Size test=60.8 || showcas... test=60.8 || showcas... - Databases and Collections: List of Databas e Collection Count test=3 || showcase=1... test=3 || showcase=1... - Databases and Collections: List of Databas e Data Size test=304 || showcase... test=304 || showcase... -
  • 31. Lukáš Korous lukas.korous@softica.eu Databases and Collections: List of Databas e Extent Count test=3 || showcase=2... test=3 || showcase=2... - Databases and Collections: List of Databas e File Size test=67108864 || sho... test=67108864 || sho... - Databases and Collections: List of Databas e Index Count test=1 || showcase=1... test=1 || showcase=1... - Databases and Collections: List of Databas e Index Size test=8176 || showcas... test=8176 || showcas... - Databases and Collections: List of Databas e Object Count test=5 || showcase=2... test=5 || showcase=2... - Databases and Collections: List of Databas e Storage Size test=20480 || showca... test=20480 || showca... - Free disk space on / 49.16 GB 50.2 GB 53.2 GB Free disk space on / (percentage) 81.41 % 83.14 % 88.11 % Host boot time 2015-09-26 17:14:27 2015-09-18 16:59:31 2015-09-18 16:59:36 Host local time 2015-11-23 17:16:37 2015-11-23 17:16:13 2015-11-23 17:16:54 Host name mongo-1 mongo-2 mongo-3 Host name of zabbix_agentd running mongo-1 mongo-2 mongo-3 Maximum number of opened files 404256 404257 404257 Maximum number of processes 32768 32768 32768 Memory: Heap Memory Size (MB) 510.74 500.73 500.86 Memory: Memory Addressing (32/64 bit) 64 64 64 Memory: Page Faults/minute 0 0 0 Memory: Resident Memory Size (MB) 679 681 682 Memory: Virtual Memory Size (MB) 11193 11179 11181 Miscellaneous: Data Collector 0 0 0 Miscellaneous: MongoDB Plugin Checksu m db987f88e0a81 2a16f59... db987f88e0a812 a16f59... db987f88e0a812 a16f59... Miscellaneous: MongoDB Plugin Data Colle ction Time (seconds) 0 0 0 Miscellaneous: MongoDB Plugin Version 0.4 0.4 0.4
  • 32. Lukáš Korous lukas.korous@softica.eu Miscellaneous: MongoDB Uptime (seconds ) 5014923 1011 5009683 Miscellaneous: MongoDB Version 3.0.6 3.0.6 3.0.6 Number of processes 89 89 89 Replication: Count of ReplicaSet Members Needing Attention 0 0 0 Replication: Entries in oplog.rs Collection 1 2792 2792 Replication: Is Mongo Server Part of a Repl icatSet Yes Yes Yes Replication: List of ReplicaSet Members in Attention State empty empty empty Replication: ReplicaSet Healthy Member C ount 3 3 3 Replication: ReplicaSet Host Names host#0 = XXX.YYY.ZZZ.... host#0 = XXX.YYY.ZZZ.... host#0 = XXX.YYY.ZZZ.... Replication: ReplicaSet Member Count 3 3 3 Replication: ReplicaSet Name initialReplicaSet initialReplicaSet initialReplicaSet Sharding: Is Mongo Server a Cluster Router (mongos process) No No No System information Linux mongo-1 3.16.0... Linux mongo-2 3.16.0... Linux mongo-3 3.16.0... System uptime 58 days, 00:53:16 66 days, 01:10:48 66 days, 01:11:24 Total disk space on / 62.96 GB 62.96 GB 62.96 GB Total memory 3.87 GB 3.87 GB 3.87 GB Total swap space 0 B 0 B 0 B Used disk space on / 11.22 GB 10.18 GB 7.18 GB Version of zabbix_agent(d) running 2.2.7 2.2.7 2.2.7 Services running on the host root@mongo-3:/home/debian# service --status-all ... [ + ] mongod (mongoDB service) ...
  • 33. Lukáš Korous lukas.korous@softica.eu [ + ] zabbix-agent (monitoring agent service) ... root@mongo-3:/home/debian#
  • 34. Lukáš Korous lukas.korous@softica.eu Ports on which MongoDB and Zabbix listen root@mongo-3:/home/debian# netstat -ap Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name ... tcp 0 0 *:zabbix-agent *:* LISTEN 8982/zabbix_agentd tcp 0 0 *:27017 *:* LISTEN 9294/mongod ... MongoDB logfile - /var/log/mongodb/mongodb.log ... 2015-11-30T09:31:23.920+0000 I ACCESS [conn465825] Successfully authenticated as principal __system on local 2015-11-30T09:31:39.905+0000 I NETWORK [conn465824] end connection XXX.YYY.ZZZ.235:33007 (1 connection now open) 2015-11-30T09:31:39.906+0000 I NETWORK [initandlisten] connection accepted from XXX.YYY.ZZZ.235:33009 #465826 (3 connections now open) 2015-11-30T09:31:40.023+0000 I ACCESS [conn465826] Successfully authenticated as principal __system on local 2015-11-30T09:31:53.942+0000 I NETWORK [conn465825] end connection XXX.YYY.ZZZ.234:60287 (1 connection now open) 2015-11-30T09:31:53.943+0000 I NETWORK [initandlisten] connection accepted from XXX.YYY.ZZZ.234:60289 #465827 (2 connections now open) 2015-11-30T09:31:53.965+0000 I ACCESS [conn465827] Successfully authenticated as principal __system on local 2015-11-30T09:31:56.422+0000 I NETWORK [initandlisten] connection accepted from 10.20.0.6:36444 #465828 (3 connections now open) 2015-11-30T09:31:56.448+0000 I ACCESS [conn465828] Successfully authenticated as principal root on admin 2015-11-30T09:31:56.457+0000 I NETWORK [conn465828] end connection 10.20.0.6:36444 (2 connections now open) ...
  • 35. Lukáš Korous lukas.korous@softica.eu Sources  wikipedia.org  mongodb.org  Softica Solutions private repositories