MongoDB World 2018: Using Change Streams to Keep Up with Your Data
3 likes1,434 views
The document discusses how change streams in MongoDB can be used to keep application data in real-time by watching for changes to documents in a collection. Change streams provide a defined API and enable scaling across nodes while ensuring total ordering of changes. They allow applications to populate an initial cache of documents and then receive notifications of inserts, updates, replaces, deletes, and invalidates in order to maintain an up-to-date cache. The document provides examples of filtering change streams and handling different operation types.
![Change Streams are Resumable
changeStream.on ('change', (change) => {
console.log(change);
cachedResumeToken = change["_id"];
});
changeStream.on ('error', () => {
if (cachedResumeToken) {
establishChangeStream (cachedResumeToken);
}
});](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-25-320.jpg)
![Change Streams Utilize the Power of the
Aggregation Framework
$match $project $addFields $replaceRoot $redact
var changeStream = coll.watch([{ $match: {operationType: {$in: ['delete', 'replace']}}}]);](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-26-320.jpg)
![Insert
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.insert({
_id: 1,
text: "hello"
})
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "insert",
"ns" : {
"db" : "test",
"coll" : "test"
},
"documentKey" : {
"_id" : 1
},
"fullDocument" : {
"_id" : 1,
"text": "hello"
},
}](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-29-320.jpg)
![Update
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.updateOne({_id: 1},
{ $set: {“text”: “updated”} })
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "update",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
},
"updateDescription" : {
"updatedFields" : {
"text" : "updated"
},
"removedFields" : [ ]
}
}](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-30-320.jpg)
![Update with fullDocument: updateLookup
> db.collection.watch([],
{ fullDocument: “updateLookup”,
maxAwaitTimeMS: 30000 }).pretty()
> db.collection.updateOne({_id: 1},
{ $set: {“number”: “5”} })
Shell 1 Shell 2
{ "_id" : (resumeToken),
"operationType" : "update",
"fullDocument" : {
"_id" : 1,
"text" : "updated",
"number" : 5
},
"ns": {"db": "test", "coll": "collection"},
"documentKey" : { "_id" : 1 },
"updateDescription" : {
"updatedFields" : { "number" : 5 },
"removedFields" : [ ]
}
}](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-31-320.jpg)
![Replace
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.replaceOne({_id: 1},
{text: "replaced"})
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "replace",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
},
"fullDocument" : {
"_id" : 1,
"text" : "replaced"
}
}](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-32-320.jpg)
![Delete
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.remove({_id: 1})
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "delete",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
}
}](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-33-320.jpg)
![Invalidate
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.drop()
Shell 1 Shell 2
{
"_id" : (token),
"operationType" : "invalidate"
}](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-34-320.jpg)
![_id: 11 (40.0, -74.0)
db.train.watch([])
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: { lat: 40.41169 }
},
documentKey: {_id: 11}
Handled
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: {
passengerCount: 5
}
}
documentKey: {_id: 11}
Handled?
App Cache _id: 11 (40.4, -74.0)
Watch for changes](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-42-320.jpg)
![db.train.watch([{
$match: {
$or: [
{'updateDescription.updatedFields.lat': {$exists: 1}},
{'updateDescription.updatedFields.lon': {$exists: 1}}
]
}
}])
_id: (resumeToken),
operationType: 'insert',
fullDocument: {
_id: 16,
lat: 40.5889, lon: -76.1938
},
documentKey: {_id: 16}
Ignored?
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: {
passengerCount: 5
}
}
documentKey: {_id: 11}
Ignored
App Cache _id: 11 (40.4, -74.0)](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-44-320.jpg)
![db.train.watch([{
$match: {
$or: [
{
operationType: 'update',
$or: [
{'updateDescription.updatedFields.lat': {$exists: 1}},
{'updateDescription.updatedFields.lon': {$exists: 1}}
]
},
{ operationType: { $in: ['delete', 'insert', 'replace'] } }
]
}
}])
App Cache _id: 11 (40.4, -74.0)
_id: (resumeToken),
operationType: 'insert',
fullDocument: {
_id: 16,
lat: 40.5889, lon: -76.1938
},
documentKey: {_id: 16}
Handled
_id: 16 (40.4, -74.0)
_id: (resumeToken),
operationType: 'delete',
documentKey: {_id: 11}Handled](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-46-320.jpg)
![find
Client Server
[ {_id: 11}…], $clusterTime: X
watch, startAtOperationTime: X
start session
startAtOperationTime](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-48-320.jpg)
![startAtOperationTime
const session = client.startSession();
const cursor = db.collection('train').find({}, { session: session });
populateApplicationCache(cursor);
const changeStream = db.collection('train').watch([], {
session: session,
startAtOperationTime: session.operationTime
});
changeStream.on('change', updateApplicationCache);](https://image.slidesharecdn.com/americasii935kevinalbertsonusingchangestreamstokeepupwithyourdata-180709191544/85/MongoDB-World-2018-Using-Change-Streams-to-Keep-Up-with-Your-Data-49-320.jpg)
MongoDB World 2018: Using Change Streams to Keep Up with Your Data
- 1. Using Change Streams to Keep Up With Your Data Kevin Albertson, Software Engineer II
- 2. Kevin Albertson Software Engineer II, MongoDB @kevinAlbs
- 5. March 8th, 2018 Point A Point B
- 8. March 8th, 2018 6:33 6:45 6:54 Tw eet:everything ok Iarrive atstation Train scheduled 7:00 Stillw aiting 7:15 Stillw aiting
- 9. March 8th, 2018 6:33 6:45 6:54 Tw eet:everything ok Iarrive atstation Train scheduled 7:15 Stillw aiting 7:00 Stillw aiting
- 12. March 8th, 2018 6:33 6:45 6:54 Tw eet:everything ok Iarrive atstation Train scheduled 8:00 A nnouncem ent 7:30 Stillw aiting 7:15 Stillw aiting 7:00 Stillw aiting 7:45 Stillw aiting
- 13. March 8th, 2018
- 15. Real-time is an Expectation
- 16. Change Streams Enable Existing Apps to Provide Real-time Data
- 17. const changeStream = db.collection('train').watch(); c h a n g e S t r e a m . o n ( ' c h a n g e ' , ( c h a n g e ) = > { c o n s o l e . l o g ( c h a n g e ) } ) ; Added in 3.6 Improved in 4.0 { "_id" : (resumeToken), "operationType" : "insert", "ns" : { "db" : "test", "coll" : "train" }, "documentKey" : { "_id" : 123 }, "fullDocument" : { "_id" : 123, "text": "hello" }, }
- 19. 1. Use Collection Access Controls 2. Present a Defined API 3. Scale Across Nodes Change Streams
- 20. Return 5 Operation Types 1. Insert 2. Update 3. Replace 4. Delete 5. Invalidate
- 21. mongos 3 2 1 Shard 1 Shard 2 Shard 3 Total Ordering of Changes Across Shards
- 22. Documents Uniquely Identified Shard key combined with _id to uniquely identify documents mongos Shard 2 Shard 3Shard 1
- 24. Change Streams are Resumable collection.watch() PP { _id: <resumeToken>, operationType: 'update' ... } SP S
- 25. Change Streams are Resumable changeStream.on ('change', (change) => { console.log(change); cachedResumeToken = change["_id"]; }); changeStream.on ('error', () => { if (cachedResumeToken) { establishChangeStream (cachedResumeToken); } });
- 26. Change Streams Utilize the Power of the Aggregation Framework $match $project $addFields $replaceRoot $redact var changeStream = coll.watch([{ $match: {operationType: {$in: ['delete', 'replace']}}}]);
- 27. 1. Collection Access Controls 2. Defined API 3. Enable Scaling 4. Total Ordering 5. Durable 6. Resumable 7. Power of Aggregation
- 28. Types of Changes
- 29. Insert > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.insert({ _id: 1, text: "hello" }) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "insert", "ns" : { "db" : "test", "coll" : "test" }, "documentKey" : { "_id" : 1 }, "fullDocument" : { "_id" : 1, "text": "hello" }, }
- 30. Update > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.updateOne({_id: 1}, { $set: {“text”: “updated”} }) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "update", "ns" : { "db" : "test", "coll" : "collection" }, "documentKey" : { "_id" : 1 }, "updateDescription" : { "updatedFields" : { "text" : "updated" }, "removedFields" : [ ] } }
- 31. Update with fullDocument: updateLookup > db.collection.watch([], { fullDocument: “updateLookup”, maxAwaitTimeMS: 30000 }).pretty() > db.collection.updateOne({_id: 1}, { $set: {“number”: “5”} }) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "update", "fullDocument" : { "_id" : 1, "text" : "updated", "number" : 5 }, "ns": {"db": "test", "coll": "collection"}, "documentKey" : { "_id" : 1 }, "updateDescription" : { "updatedFields" : { "number" : 5 }, "removedFields" : [ ] } }
- 32. Replace > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.replaceOne({_id: 1}, {text: "replaced"}) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "replace", "ns" : { "db" : "test", "coll" : "collection" }, "documentKey" : { "_id" : 1 }, "fullDocument" : { "_id" : 1, "text" : "replaced" } }
- 33. Delete > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.remove({_id: 1}) Shell 1 Shell 2 { "_id" : (resumeToken), "operationType" : "delete", "ns" : { "db" : "test", "coll" : "collection" }, "documentKey" : { "_id" : 1 } }
- 34. Invalidate > db.collection.watch([], {maxAwaitTimeMS: 30000}).pretty() > db.collection.drop() Shell 1 Shell 2 { "_id" : (token), "operationType" : "invalidate" }
- 35. New in 4.0
- 36. 1. startAtOperationTime 2. Change stream on db or client 3. Resumable before first change
- 37. Use Case
- 38. Real-time Train Arrival Estimates
- 39. Collection > db.train.find().pretty() { "_id" : 11, "position" : { "lat" : 40.04714, "lon" : -74.0351 } }
- 40. App Caches State of All Trains App Cache
- 41. _id: 11 (40.0, -74.0) db.train.find().readConcern("majority") App Cache Populate the Cache
- 42. _id: 11 (40.0, -74.0) db.train.watch([]) _id: (resumeToken), operationType: 'update', updateDescription: { updatedFields: { lat: 40.41169 } }, documentKey: {_id: 11} Handled _id: (resumeToken), operationType: 'update', updateDescription: { updatedFields: { passengerCount: 5 } } documentKey: {_id: 11} Handled? App Cache _id: 11 (40.4, -74.0) Watch for changes
- 43. $match to Filter Irrelevant Changes
- 45. Handle All Document Changes If Caching State of Documents:
- 46. db.train.watch([{ $match: { $or: [ { operationType: 'update', $or: [ {'updateDescription.updatedFields.lat': {$exists: 1}}, {'updateDescription.updatedFields.lon': {$exists: 1}} ] }, { operationType: { $in: ['delete', 'insert', 'replace'] } } ] } }]) App Cache _id: 11 (40.4, -74.0) _id: (resumeToken), operationType: 'insert', fullDocument: { _id: 16, lat: 40.5889, lon: -76.1938 }, documentKey: {_id: 16} Handled _id: 16 (40.4, -74.0) _id: (resumeToken), operationType: 'delete', documentKey: {_id: 11}Handled
- 47. Race Between Populating and Listening! db.train.find().readConcern("majority") . . . db.train.watch() update occurs!
- 48. find Client Server [ {_id: 11}…], $clusterTime: X watch, startAtOperationTime: X start session startAtOperationTime
- 49. startAtOperationTime const session = client.startSession(); const cursor = db.collection('train').find({}, { session: session }); populateApplicationCache(cursor); const changeStream = db.collection('train').watch([], { session: session, startAtOperationTime: session.operationTime }); changeStream.on('change', updateApplicationCache);
- 50. App Complete! 1. Doesn't miss updates 2. Handles network blips 3. Reduces bandwidth 4. Easy to prototype