[MongoDB.local Bengaluru 2018] Using Change Streams to Keep Up With Your Data

#MDBlocal
Using Change Streams to Keep
Up With Your Data
24th April, 2018 | Bengaluru

#MDBlocal
Ankit Kakkar
Technical Services Manager | MongoDB

#MDBlocal
1. Change Stream Characteristics
2. Operation Types
3. Development Tips
4. Underlying Technology
AGENDA

#MDBlocal
REAL-TIME IS AN EXPECTATION

#MDBlocal
INTRODUCING: CHANGE STREAMS
Allows you to watch all the changes against a given collection.
CHANGESTREAMSAPI
Event Notifications
● Application
● MongoDB Stitch
● Message Queue

#MDBlocal
CHANGE STREAMS: USE CASES
● Refreshing trading apps as
stock prices change
● Syncing changes across
microservices
● Updating dashboards,
analytics systems, search
engines
● IOT data pipelines - generating
alarms in response to connected
asset failures
● Push new credit card transaction
into Machine Learning models to
recalculate risk
● Maintaining multiplayer game
scoreboards.

#MDBlocal
MongoD
B
Action
Handle
r

#MDBlocal
changeStream = db.collection(“foo”).watch();
changeStream.on(“change”, function(change){
console.log(change)
});

#MDBlocal
CHANGE STREAM
CHARACTERISTICS

#MDBlocal
CHANGE STREAMS UTILISE
COLLECTION ACCESS CONTROLS,
PRESENT A DEFINED API, AND
ENABLE SCALING ACROSS
PRIMARIES AND SECONDARIES.

#MDBlocal
SHARD
A
SHARD
C
SHARD
B
mongos
TOTAL ORDERING OF CHANGES
ACROSS SHARDS

#MDBlocal
SHARD
A
SHARD
C
SHARD
B
mongos
TOTAL ORDERING OF CHANGES
ACROSS SHARDS
1 2 3

#MDBlocal
XPRIMARY
PRIMARY
CHANGES ARE DURABLE
SECONDARY
SECONDARY

#MDBlocal
X
PRIMARY
CHANGES ARE RESUMABLE
SECONDARY
{_id: <resumeToken>,
operationType: ‘update’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘room12345’},
updateDescription: {updatedFields: {temperature: 27},
removedFields: [‘humidity’]}
}
SECONDARY
var newChangeStream = coll.watch({
resumeAfter: <cachedResumeToken>
});

#MDBlocal
CHANGE STREAMS UTILISE THE POWER
OF THE AGGREGATION FRAMEWORK
$match $project $addFields $replaceRoot $redact
var changeStream = coll.watch([
{$match: {$or: [{operationType: 'delete' },
{operationType:'replace'}]}}
]);

#MDBlocal
1. COLLECTION ACCESS CONTROLS
2. DEFINED API
3. ENABLE SCALING
4. TOTAL ORDERING
5. DURABLE
6. RESUMABLE
7. POWER OF AGGREGATION

#MDBlocal
5 OPERATION TYPES
DELETE
INSERT
REPLACE
UPDATE
INVALIDATE

#MDBlocal
{ _id: (resumeToken),
operationType: ‘insert’,
ns: {db: ‘test’, coll: ‘foo’},
documentKey: {userName: ‘anish123’,
_id: ObjectId("58a4eb4a30c75625e00d2820")},
fullDocument: {_id: ObjectId("58a4eb4a30c75625e00d2820"),
userName: ‘anish123’,
name: ‘Anish Bhanwala’}
}
5 OPERATION TYPES: INSERT

#MDBlocal
DEFINING THE documentKey
{ _id: (resumeToken),
documentKey: {userName: ‘anish123’,
userName: ‘anish123’,
name: ‘Anish Bhanwala’}
}

#MDBlocal
DEFINING THE documentKey
The “_id” field of the document created or modified by
the operation.
● Replica Set : ‘_id’
● Sharded Cluster : ‘_id’ + ‘shard key’

#MDBlocal
{_id: (resumeToken),
documentKey: {userName: ‘neerajchopra’,
updateDescription: {updatedFields: {email: ‘neeraj@cwg’},
removedFields: [‘phoneNumber’]}
}
5 OPERATION TYPES: UPDATE

#MDBlocal
UPDATE
changeStream = coll.watch([], {fullDocument:'updateLookup'});
removedFields: [‘phoneNumber’]}
}

#MDBlocal
UPDATE
fullDocument
var changeStream = coll.watch( {fullDocument:'updateLookup'} );
removedFields: [‘phoneNumber’]},
fullDocument: { _id: ObjectId("58a4eb4a30c75625e00d2820"),
name: ‘Neeraj Chopra’,
userName: ‘neerajchopra’,
email: ‘neeraj@cwg’,
team: ‘replication’}
}

#MDBlocal
operationType: ‘replace’,
documentKey: {userName: ‘manika123’,
_id:
ObjectId("58a4eb4a30c75625e00d2820")},
userName: ‘manika123’,
name: ‘Manika Batra’}
}
5 OPERATION TYPES: REPLACE

#MDBlocal
operationType: ‘delete’,
documentKey: {userName: ‘alice123’,
_id:
ObjectId("58a4eb4a30c75625e00d2820")
}
}
5 OPERATION TYPES: DELETE

#MDBlocal
5 OPERATION TYPES: INVALIDATE
operationType: ‘invalidate’,
ns: {db: ‘test’, coll: ‘foo’}
}

#MDBlocal
Action Handler
var changeStream = coll.watch(
[{ $match: { 'fullDocument.temperature': { $gte:27 }}],
{ fullDocument: 'updateLookup' });
documentKey: {_id: ‘LivingRoomGreen’},
fullDocument: {_id: ‘LivingRoomGreen’),
temperature: 30,
username: ‘ankit_kakkar’}
}
temperature: 30
username: ankit_kakkar

#MDBlocal
Action Handler
documentKey : {_id: ‘OrangeRoomC’
fullDocument: {_id: ‘OrangeRoomC’,
temperature: 29,
username: ‘Ankur_raina’ }
}
temperature: 30

#MDBlocal
Action Handler
documentKey: {_id: ‘LivingRoomGreen’)},
updateDescription: {updatedFields: {temperature: 25}
fullDocument: {_id: ‘LivingRoomGreen’,
temperature: 25,
}
temperature: 30

#MDBlocal
IF YOUR APPLICATION REQUIRES STATE
ALWAYS MATCH ON UNCHANGING
FIELDS

#MDBlocal
Action Handler
[{ $match: { 'fullDocument.username: ‘ankit_kakkar’ }],
documentKey: {_id: ‘blueRoomB’},
fullDocument: {_id: ‘blueRoomB’,
temperature: 32,
}
temperature: 30
temperature: 32

#MDBlocal
Action Handler
documentKey: {_id: ‘blueRoomB’}
}
temperature: 30
temperature: 32

#MDBlocal
Action Handler
documentKey : {_id: ‘blueRoomB’},
fullDocument: {_id: ‘blueRoomB’,
temperature: 29,
username: ‘nishant’}
}
temperature: 30
temperature: 32

#MDBlocal
IF YOUR APPLICATION NEEDS TO NOTIFY ON
DELETED DATA
ALWAYS HANDLE DELETES AND
REPLACES APPROPRIATELY

#MDBlocal
IF YOU NEED TO SEE EVERY CHANGE (EVEN
OUTDATED CHANGES) TO A DOCUMENT
HAVE MATCHING FIELDS IN THE
DOCUMENT KEY

#MDBlocal
Action Handler
[{ $match: { ‘documentKey._id.username’: ‘ankit_kakkar’}],
documentKey: {_id: {roomId: ‘blueRoomB’,
username: ‘ankit_kakkar’}},
fullDocument: {_id: {roomId:‘blueRoomB’,
username:’ankit_kakkar’},
temperature: 32}
}
temperature: 30
temperature: 32

#MDBlocal
Action Handler
documentKey: {_id: {roomId: ‘blueRoomB’,
username:
‘ankit_kakkar’}}
}
temperature: 30
temperature: 32

#MDBlocal
Action Handler
documentKey: {_id: {roomId:‘LivingRoomGreen’,
username: ‘ankit_kakkar’}},
fullDocument: {_id: {roomId: ‘LivingRoomGreen’,
username: ‘nishant’},
temperature: 26}
}
temperature: 30

#MDBlocal
1. If your application requires state:
Match on unchanging fields.
2. If your application needs to notify on deleted data:
Handle deletes and replaces appropriately.
3. If your application needs to see every change
(even outdated changes):
Have matching fields in the document key.

#MDBlocal
UNDERLYING
TECHNOLOGY

#MDBlocal
The secondaries replicate the primary’s
oplog and apply the operations to their
data sets such that the secondaries’ data
sets reflect the primary’s data set.
REPLICATION

#MDBlocal
4 3 2 15
CAPPED COLLECTION (OPLOG)

#MDBlocal
5 4 3 26
CAPPED COLLECTION (OPLOG)

#MDBlocal
7 6 5 4 3 2 1
CACHED RESUME TOKEN : 3
NEWEST OLDEST
DELETED
RESUMABLE

#MDBlocal
7 6 5 4 3 2 1
CACHED RESUME TOKEN : 1
NEWEST OLDEST
DELETED
NON-RESUMABLE

#MDBlocal
HOW TO GET STARTED
● MongoDB v3.6 (WiredTiger)
● Replica Set (Data bearing node)

[MongoDB.local Bengaluru 2018] Using Change Streams to Keep Up With Your Data

More Related Content

What's hot (20)

Similar to [MongoDB.local Bengaluru 2018] Using Change Streams to Keep Up With Your Data (20)

More from MongoDB (20)

Recently uploaded (20)

[MongoDB.local Bengaluru 2018] Using Change Streams to Keep Up With Your Data