Massively Distributed Backups at Facebook Scale - Shlomo Priymak, Facebook - DevOpsDays Tel Aviv 2015
2 likes733 views
1) Facebook uses differential backups to backup their MySQL databases at massive scale. Differential backups capture only the changes between full backups, reducing storage and backup time requirements compared to full backups alone. 2) Differential backups work by comparing the data in the previous full backup to the current full backup and identifying inserted, deleted, and updated rows. The changes are stored in the differential backup. 3) Facebook distributes backups across data centers globally to balance load and minimize cross-region traffic. A hashing algorithm distributes database shards into buckets that are then assigned proportionally to HDFS clusters based on size.
![Differential Backup Generation
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Santa Clara`),
(400, ‘Los Angeles’),
[...]
);
INSERT INTO t1 VALUES (
);
INSERT INTO t1 VALUES (
);
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Oakland`),
(3, ‘Menlo Park’),
[...]
);
No Change
Inserted Rows
CREATE TABLE t (id int,
city char(50);
/* ORDERING KEY : (id) */
Full Backup (old) Full Backup (new)
Deleted Rows](https://image.slidesharecdn.com/shlomo-massivelydistributedbackupsatfacebookscale-151012084557-lva1-app6891/85/Massively-Distributed-Backups-at-Facebook-Scale-Shlomo-Priymak-Facebook-DevOpsDays-Tel-Aviv-2015-18-320.jpg)
![Differential Backup Generation
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Santa Clara`),
(400, ‘Los Angeles’),
[...]
);
INSERT INTO t1 VALUES (
(2, ‘Santa Clara’),
);
INSERT INTO t1 VALUES (
(2, ‘OakLand’),
);
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Oakland`),
(3, ‘Menlo Park’),
[...]
);
Row Updated
Inserted RowsDeleted Rows
Full Backup (old) Full Backup (new)
CREATE TABLE t (id int,
city char(50);
/* ORDERING KEY : (id) */](https://image.slidesharecdn.com/shlomo-massivelydistributedbackupsatfacebookscale-151012084557-lva1-app6891/85/Massively-Distributed-Backups-at-Facebook-Scale-Shlomo-Priymak-Facebook-DevOpsDays-Tel-Aviv-2015-19-320.jpg)
![Differential Backup Generation
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Santa Clara`),
(400, ‘Los Angeles’),
[...]
);
INSERT INTO t1 VALUES (
(2, ‘Santa Clara’),
);
INSERT INTO t1 VALUES (
(2, ‘OakLand’),
(3, ‘Menlo Park’),
);
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Oakland`),
(3, ‘Menlo Park’),
[...]
);
Row Deleted
Inserted RowsDeleted Rows
Full Backup (old) Full Backup (new)
CREATE TABLE t (id int,
city char(50);
/* ORDERING KEY : (id) */](https://image.slidesharecdn.com/shlomo-massivelydistributedbackupsatfacebookscale-151012084557-lva1-app6891/85/Massively-Distributed-Backups-at-Facebook-Scale-Shlomo-Priymak-Facebook-DevOpsDays-Tel-Aviv-2015-20-320.jpg)
![Differential Backup Generation
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Santa Clara`),
(400, ‘Los Angeles’),
[...]
);
INSERT INTO t1 VALUES (
(2, ‘Santa Clara’),
(400, ‘Los Angeles’),
);
INSERT INTO t1 VALUES (
(2, ‘OakLand’),
(3, ‘Menlo Park’),
);
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Oakland`),
(3, ‘Menlo Park’),
[...]
);
Row Inserted
Inserted RowsDeleted Rows
Full Backup (old) Full Backup (new)
CREATE TABLE t (id int,
city char(50);
/* ORDERING KEY : (id) */](https://image.slidesharecdn.com/shlomo-massivelydistributedbackupsatfacebookscale-151012084557-lva1-app6891/85/Massively-Distributed-Backups-at-Facebook-Scale-Shlomo-Priymak-Facebook-DevOpsDays-Tel-Aviv-2015-21-320.jpg)
![Restoring Diff Backup
INSERT INTO t1 VALUES (
(2, ‘Santa Clara’),
(400, ‘Los Angeles’),
);
Inserted Rows
INSERT INTO t1 VALUES (
(2, ‘OakLand’),
(3, ‘Menlo Park’),
);
Deleted Rows
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Oakland`),
(3, ‘Menlo Park’),
[...]
);
Full Backup (old)
3-Way Merge
INSERT INTO t VALUES
(1, ‘San Fransisco’),
(2, ‘Santa Clara`),
(400, ‘Los Angeles’),
[...]
);
Full Backup (new)](https://image.slidesharecdn.com/shlomo-massivelydistributedbackupsatfacebookscale-151012084557-lva1-app6891/85/Massively-Distributed-Backups-at-Facebook-Scale-Shlomo-Priymak-Facebook-DevOpsDays-Tel-Aviv-2015-23-320.jpg)
Massively Distributed Backups at Facebook Scale - Shlomo Priymak, Facebook - DevOpsDays Tel Aviv 2015
- 2. Massively Distributed Backup at Facebook Scale Shlomo Priymak (shlomo@fb.com, @shlomoid) Production Engineering Manager, MySQL Infrastructure
- 4. Sharding fbid is 64bit integer map(fbid) = shard id
- 5. { "id": "101231234567123", "name": "Shlomo Priymak" } graph API Example graph.facebook.com/me
- 6. server instance shard #4 shard #3 shard #2 shard #1
- 7. server instance shard #4 shard #3 shard #2 shard #1 server instance shard #4 shard #3 shard #2 shard #1 server instance shard #4 shard #3 shard #2 shard #1 Master Slaves Replica Set
- 8. Prineville, Oregon Altoona, Iowa Forest City, North Carolina Ashburn,Virginia Luleå, Sweden
- 9. 100
- 10. 1000+
- 14. • `mysqldump` • --single-transaction • Logical Read Ahead Full Dumps
- 15. Logical vs. Physical Logical Physical External Tools Yes No Size Small Large Single Table Restore Easy Difficult Debug Corruption Easy Difficult Compressibility Excellent Meh Backup / Restore Duration Long Short
- 17. Differential Backup 0 2 4 6 8 % of space taken by differential backups Day 1 Day 2 Day 3 Day 4 0 25 50 75 100 Relative backup space usage Day 1 Day 2 Day 3 Day 4 Full Backup Differential Backup
- 18. Differential Backup Generation INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Santa Clara`), (400, ‘Los Angeles’), [...] ); INSERT INTO t1 VALUES ( ); INSERT INTO t1 VALUES ( ); INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Oakland`), (3, ‘Menlo Park’), [...] ); No Change Inserted Rows CREATE TABLE t (id int, city char(50); /* ORDERING KEY : (id) */ Full Backup (old) Full Backup (new) Deleted Rows
- 19. Differential Backup Generation INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Santa Clara`), (400, ‘Los Angeles’), [...] ); INSERT INTO t1 VALUES ( (2, ‘Santa Clara’), ); INSERT INTO t1 VALUES ( (2, ‘OakLand’), ); INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Oakland`), (3, ‘Menlo Park’), [...] ); Row Updated Inserted RowsDeleted Rows Full Backup (old) Full Backup (new) CREATE TABLE t (id int, city char(50); /* ORDERING KEY : (id) */
- 20. Differential Backup Generation INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Santa Clara`), (400, ‘Los Angeles’), [...] ); INSERT INTO t1 VALUES ( (2, ‘Santa Clara’), ); INSERT INTO t1 VALUES ( (2, ‘OakLand’), (3, ‘Menlo Park’), ); INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Oakland`), (3, ‘Menlo Park’), [...] ); Row Deleted Inserted RowsDeleted Rows Full Backup (old) Full Backup (new) CREATE TABLE t (id int, city char(50); /* ORDERING KEY : (id) */
- 21. Differential Backup Generation INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Santa Clara`), (400, ‘Los Angeles’), [...] ); INSERT INTO t1 VALUES ( (2, ‘Santa Clara’), (400, ‘Los Angeles’), ); INSERT INTO t1 VALUES ( (2, ‘OakLand’), (3, ‘Menlo Park’), ); INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Oakland`), (3, ‘Menlo Park’), [...] ); Row Inserted Inserted RowsDeleted Rows Full Backup (old) Full Backup (new) CREATE TABLE t (id int, city char(50); /* ORDERING KEY : (id) */
- 22. Final Output INSERT INTO t1 VALUES ( (2, ‘Santa Clara’), (400, ‘Los Angeles’), ); INSERT INTO t1 VALUES ( (2, ‘OakLand’), (3, ‘Menlo Park’), ); Inserted RowsDeleted Rows
- 23. Restoring Diff Backup INSERT INTO t1 VALUES ( (2, ‘Santa Clara’), (400, ‘Los Angeles’), ); Inserted Rows INSERT INTO t1 VALUES ( (2, ‘OakLand’), (3, ‘Menlo Park’), ); Deleted Rows INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Oakland`), (3, ‘Menlo Park’), [...] ); Full Backup (old) 3-Way Merge INSERT INTO t VALUES (1, ‘San Fransisco’), (2, ‘Santa Clara`), (400, ‘Los Angeles’), [...] ); Full Backup (new)
- 24. • Point in time recovery • Global Transaction IDs Binary Logs
- 26. • Everything, Every Day • Streaming Binary Logs • Multiple stages • HDFS • Offsite Backup Schedule What, When, Where
- 27. Backup Schedule Full, Diff, Diff, Diff, Full, Diff, Diff, Diff, Full, Diff, Diff, Diff… Full 5 Diff 6 Diff 7 Diff 8 Full 9 Diff 10 Diff 11 Diff 12 Full 1 Diff 2 Diff 3 Diff 4
- 28. Backup Traffic 3.5 Tb/s Peak ~0.5 Tb/s Trough
- 29. Differential Backup Stages 1) mysql HDFSFull (new) 2) HDFS Full (new) Full (old) Diff (new) HDFSDiffer
- 30. • Too much HDFS I/O • Too much network I/O • Too long #fail
- 31. Differential Backup Stages 1) mysql HDFSFull (new) 2) HDFS Full (new) Full (old) Diff (new) HDFSDiffer
- 32. Differential Streaming mysql HDFS Full(new) Full(old) Diff (new)Differ HDFS Database Server
- 33. Prineville, Oregon Altoona, Iowa Forest City, North Carolina Ashburn,Virginia Luleå, Sweden
- 35. • Equalize HDFS cluster usage • Minimize cross-region traffic • Avoid broken replicas • Consistency • Backup at least once! System Design Goals
- 36. • Define allocation globally • Hash shards into a 1000 buckets • Allocate buckets to clusters, proportional to size Distribution Algorithm
- 37. server instance shard 14 shard 13 shard 12 shard 11 shard 10 shard 9 shard 8 shard 7 shard 6 shard 5 shard 4 shard 3 shard 2 shard 1
- 38. 998997 999 4 1000 321 shard 14 shard 13shard 12 shard 11 shard 10 shard 9 shard 8 shard 7 shard 6 shard 5 shard 4 shard 3 shard 2 shard 1 1000 Buckets
- 39. HDFS 2HDFS 1 HDFS 3 HDFS 4 HDFS 5 101 400100 401 600 601 850 851 1000 1 PB 3 PB 2 PB 2.5 PB 1.5 PB 1 100 buckets 300 buckets 200 buckets 250 buckets 150 buckets Total buckets: 1000 Total size: 10 PB (Example)
- 40. HDFS 2HDFS 1 HDFS 3 HDFS 4 HDFS 5 101 400100 401 600 601 850 851 1000 1 PB 3 PB 2 PB 2.5 PB 1.5 PB Bucket 20 Bucket 200 Bucket 500 Bucket 650 Bucket 900 Bucket 30 Bucket 400 1 Bucket 700 Bucket 800
- 41. Unified Pool
- 42. Some are More Equal than Others?
- 43. sorted unsorted
- 44. sorted unsorted
- 46. A CB 1. C 2. A 3. B 1. C 2. A 3. B 1. C 2. A 3. B Is Alive?
- 47. A CB α HDFS ɣ HDFS Δ(B, α) = 10 Δ(A, α) = 0 Δ(A, ɣ) = 20 Δ(B, ɣ) = 15 Δ(C, ɣ) = 0 HDFS Priority ɣ 0 α 1
- 48. rank(mysql, hdfs) ≡ ( Δ(mysql, hdfs), priority(hdfs), )
- 49. 𝕄 =｛A, B, C｝// MySQL Servers 𝓗 =｛α, ɣ｝// HDFS Clusters 𝕄×𝓗 =｛(m, h): m∈𝕄 ∧ h∈𝓗｝ return sort(𝕄×𝓗, key=rank(m, h))
- 50. Δ priority A, α 0 1 A, ɣ 20 0 B, α 10 1 B, ɣ 15 0 C, α 20 1 C, ɣ 0 0
- 51. Δ priority C, ɣ 0 0 A, α 0 1 B, α 10 1 B, ɣ 15 0 A, ɣ 20 0 C, α 20 1
- 52. Pile it Up A CB
- 53. War Story: Cluster Turn Up
- 54. • New HDFS Cluster • New Datacenter • Slow Ramp-Up New Region Cluster Turn Up
- 55. • Network woes • Pulling full backups to create diffs! • Fix: run full when target HDFS changes New Region Cluster Turn Up
- 56. New Region Cluster Turn Up, cont.
- 57. New Region Cluster Turn Up, cont.
- 58. Call From the Engine Room
- 59. • Emergency meeting • Fix: Turn off a few racks Divert Power to the Shields!
- 60. The Future!
- 61. • Record previous value by default • Binary Logs + Binary Logs => Diff • Full + Diff => Full • In theory, run full backup only once! Row Based Binary Logs