Bucardo
6 likes3,145 views
The document provides an overview of Bucardo, a replication system for PostgreSQL that enables various methods of data synchronization, including master-slave and master-master configurations. It details the setup process, system requirements, strengths, and limitations, while also covering internal structures, sync processes, and administrative tasks. Additionally, it discusses error handling, hooks for custom code, and resources for further development and support.
![Setting up
• Initiate a one-time-copy: onetimecopy=[1|2]
• Set up ongoing syncs: enabled by default
(can set timeouts)
• MANY configuration options
en
dp
oi
nt
.c
om](https://image.slidesharecdn.com/bucardo-091017075408-phpapp02/85/Bucardo-25-320.jpg)
![Triggers
INSERT, UPDATE, DELETE
bucardo_add_delta
IN
SE
RT
bucardo_delta
bucardo_triggerkick_[sync]
NOTIFY!
en
dp
oi
nt
.c om](https://image.slidesharecdn.com/bucardo-091017075408-phpapp02/85/Bucardo-39-320.jpg)
Bucardo
- 1. Bucardo: Replication with tiny little goats Selena Deckelmann selena@endpoint.com en dp oi nt .c om http://www.flickr.com/photos/kevincollins/
- 2. Hi! ★ Software Engineer / Postgres Lead http://endpoint.com ★ We’re hiring! http://tr.im/EMGj en dp oi nt .c om
- 3. en dp http://www.telegraph.co.uk/science/science-news/4409958/Extinct-ibex-is- oi nt .c resurrected-by-cloning.html om
- 4. What can it do? en dp oi nt .c http://www.flickr.com/photos/8602783@N06/2294028540 om
- 5. Master-slave • Scaling/Redundancy • UPGRADES (8.x -> 8.4) • Reporting databases • Data warehousing • Now handles SEQUENCES! en dp oi nt .c om
- 6. Master-master • Write between both databases! • Sync type: SWAP • Conflict resolution en dp oi nt .c om
- 7. Custom code • Respond to change • Filter • Run arbitrary code! en dp oi nt .c om
- 8. STORY TIME! item ski item ski item ski item item ski item ski item onhand ski item skis 10 ski item ski item ski item ski on hand = count(item) en dp oi nt .c om
- 9. prod db reporting db new! item NOTIFY bucardo ski sync B new! item syn ski cC on hand cA Y TIF +1 syn NO custom code count(items) and other stuff en dp oi nt .c om goat courtesy of: http://www.flickr.com/photos/nuritwilde/394603312
- 10. Strengths • Drop-in, no changes to Postgres • Control daemon/DB can run anywhere • EXTREME async • No locking of slaves • Command-line controls rock! en dp oi nt .c om
- 11. The Fine Print • No smooth DDL handling • Not for failover (aka high availability) • bloat: pg_listener, q, bucardo_delta, bucardo_track • No locking of slaves • No Windows support • One developer en dp oi nt .c om http://www.flickr.com/photos/ncarey/135901120/
- 12. en dp oi nt .c http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling om
- 13. Bucardo Concepts en dp oi nt .c om http://www.flickr.com/photos/kendrick/361173694
- 14. Glossary of terms • replication: copying data from one place to another • DDL: Data Definition Language (CREATE/ DROP/ALTER) • DML: Data Manipulation Language (SELECT/ INSERT/UPDATE/DELETE) • triggers: bits of code run before or after DML en dp oi nt .c om
- 15. Glossary of terms • goat: a database object (table or sequence) • herd: table contains unique name for a group of goats • syncs: replication events (one herd per sync) en dp oi nt .c om
- 16. flexible architecture bucardo control database master DB slave DB en dp oi nt .c om
- 17. flexible architecture same db bucardo control cluster database master DB slave DB en dp oi nt .c om
- 18. flexible architecture same db cluster bucardo control database master DB slave DB en dp oi nt .c om
- 19. flexible architecture same db cluster bucardo control database master DB slave DB en dp oi nt .c om
- 20. Installing en dp oi nt .c om http://www.flickr.com/photos/kendrick/361173694
- 21. Bucardo Requirements • Postgres (8.3 or higher recommended) • Perl & some packages: • DBD::Pg 2.0 • DBI 1.51 • DBIx::Safe 1.2.4 • ExtUtils::MakeMaker 6.32 • pl/perlu and pl/pgsql en dp oi nt .c om
- 22. Simple set up BUCARDO CONTROL database setup: bucardo_ctl install en dp oi nt .c om
- 23. Simple set up MASTER setup: bucardo_ctl add db MyDB name=master bucardo_ctl add all tables herd=all_tables bucardo_ctl add all sequences herd=all_tables SLAVE setup: bucardo_ctl add db MyDB name=slave port=6543 en dp oi nt .c om
- 24. Setting up, cont. TEST: bucardo_ctl validate all REPLICATE: bucardo_ctl add sync delta type=pushdelta source=all_tables targetdb=slave bucardo_ctl start (installs triggers!) en dp oi nt .c om
- 25. Setting up • Initiate a one-time-copy: onetimecopy=[1|2] • Set up ongoing syncs: enabled by default (can set timeouts) • MANY configuration options en dp oi nt .c om
- 26. Admin stuff • Can set up defaults in .bucardorc • Only need one ‘bucardo’ database • Uses the ‘bucardo’ schema in replication sets for ease of admin, removal en dp oi nt .c om
- 27. Smooth upgrades! bucardo_ctl upgrade • Stop bucardo first (bucardo_ctl stop) • applies DDL changes, functions • see also UPGRADE file en dp oi nt .c om
- 28. Stuff to watch out for • search_path for ‘bucardo’ user • PL/Perlu & pl/pgsql must be installed • Uses prepared statements, turn off with: bucardo_ctl update db MyDB server_side_prepares=0 en dp oi nt .c om
- 29. LIVE DEMO! • http://github.com/selenamarie/ bucardo_examples en dp oi nt .c om
- 30. Internals en dp oi nt http://www.flickr.com/photos/chelseagirlphotos .c om
- 31. Internals bucardo=# d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+--------- bucardo | audit_pid | table | bucardo bucardo | audit_pid_id_seq | sequence | bucardo bucardo | bucardo_config | table | bucardo bucardo | bucardo_custom_trigger | table | bucardo bucardo | bucardo_custom_trigger_id_seq | sequence | bucardo bucardo | bucardo_log_message | table | bucardo bucardo | bucardo_rate | table | bucardo bucardo | customcode | table | bucardo bucardo | customcode_id_seq | sequence | bucardo bucardo | customcode_map | table | bucardo bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo en (21 rows) dp oi nt .c om
- 32. goats Table "bucardo.goat" Column | Type | ----------------------+--------------------------+ id | integer | db | text | schemaname | text | tablename | text | reltype | text | pkey | text | qpkey | text | pkeytype | text | has_delta | boolean | ping | boolean | ... en dp oi nt .c om
- 33. goats # bucardo_ctl list tables Table: archive.course_logging DB: odw_master PK: none Table: archive.messages DB: odw_master PK: none Table: archive.student_assignments DB: odw_master PK: none Table: archive.student_courses DB: odw_master PK: none Table: logging.activity DB: odw_master PK: activity_name (varchar) Table: logging.course_logging DB: odw_master PK: course_logging_id (int4) Table: public.ecash_transaction DB: odw_master PK: trans_id (int4) Table: public.feedback DB: odw_master PK: date (timestamp) Table: public.holidays DB: odw_master PK: customernum|month|day| year (int4|int4|int4|int4) ... en dp oi nt .c om
- 34. bucardo_ctl list sequences Sequence 63: logging.logging_seq Sequence 64: public.trans_id_seq Sequence 65: public.num_seq Sequence 66: public.upload_seq en dp oi nt .c om
- 35. bucardo database bucardo=# d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+--------- ... bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | q | table | bucardo bucardo | sync | table | bucardo bucardo | upgrade_log | table | bucardo (21 rows) en dp oi nt .c om
- 36. Status # bucardo_ctl status Days back: 3 User: bucardo Database: bucardo Port: 8000 PID of Bucardo MCP: 6876 Name Type State PID Last_good Time I/U/D Last_bad Time ========+=====+=====+====+=========+=====+=====+========+==== a_delta | P |idle |6886|1m4s |0s |0/0/0|unknown | b_delta | P |idle |6885|1m4s |0s |0/0/0|9h32m47s|0s c_delta | P |idle |6887|1m4s |0s |0/0/0|unknown | d_delta | P |idle |6888|54s |0s |0/0/0|2m6s |3s e_delta | P |idle |6890|1m4s |0s |0/0/0|9h32m47s|0s f_delta | P |idle |6889|1m4s |0s |0/0/0|9h32m47s|0s en dp oi nt .c om
- 37. # bucardo_ctl status a_delta Days back: 3 User: bucardo Database: bucardo Port: 8000 ====================================================================== Sync name: a_delta Current state: idle (PID = 6888) Type: pushdelta Source herd/database: all_a / a_master Target database: a_slave Tables in sync: 26 Last good: 3m 34s (time to run: 0s) Last good time: Oct 16, 2009 17:52:20 Target: a_slave Ins/Upd/Del: 20 / 0 / 50 Last bad: 4m 47s (time to run: 3s) Last bad time: Oct 16, 2009 17:51:08 Target: a_slave Latest bad reason: ? PID file: /var/run/bucardo/bucardo.ctl.sync.a_delta.pid PID file created: Fri Oct 16 17:52:09 2009 Status: active Limitdbs: 0 Priority: 0 Checktime: none Overdue time: 00:00:00 Expired time: 00:00:00 Stayalive: yes Kidsalive: yes Rebuild index: 0 Do_listen: yes Ping: yes Makedelta: no en dp Onetimecopy: 0 oi nt .c om
- 38. How Bucardo replicates en dp oi nt .c om
- 39. Triggers INSERT, UPDATE, DELETE bucardo_add_delta IN SE RT bucardo_delta bucardo_triggerkick_[sync] NOTIFY! en dp oi nt .c om
- 40. NOTIFY / LISTEN • Simple interprocess communication psql# LISTEN scotch; psql# NOTIFY scotch; Asynchronous notification "scotch" received from server process with PID 1337. • Any Postgres process can listen in en dp oi nt .c om
- 41. Actual Triggers! Table "public.test" Column | Type | Modifiers --------+-----------------------+----------- test | character varying(15) | not null Indexes: "test_pkey" PRIMARY KEY, btree ("test") Triggers: bucardo_add_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH ROW EXECUTE PROCEDURE bucardo_add_delta_role() bucardo_triggerkick_a_delta AFTER INSERT OR DELETE OR UPDATE ON test FOR EACH STATEMENT EXECUTE PROCEDURE bucardo_triggerkick_a_delta() en dp oi nt .c om
- 42. Bucardo NOTIFY NOTIFY NOTIFY MCP CTL KID TRIGGER INSERT INTO “q” SELECT from “q” UPDATE “q” Flow of control REPLICATION en EVENT! dp oi nt .c om
- 43. How Syncs happen • NOTIFY vs. timeout vs. kick • NOTIFY - from master db to MCP • timeout - controller detects • kick - manual/from MCP to controller en dp oi nt .c om
- 44. Processes! • MCP, CTL, KID • MCP: master control process • CTL: controller (track, kick off/kill KIDs) • KID: sync processes en dp oi nt .c om
- 45. Administrativa • Totally ok to kill KIDs • CTL cleans up after you • May need to kill open database handles (for unfinished COPYs, etc) en dp oi nt .c om
- 46. Other interesting tables On your master DB: • bucardo_delta table has: ID, txn_time • bucardo_track table: who’s done what? en dp oi nt .c om
- 47. Sync types • Fullcopy. Timeout/kick. TRUNCATE or DELETE, then COPY • Pushdelta. Trigger. One way master-slave. • Swap. Trigger. Two-way. Conflict resolution. en dp oi nt .c om
- 48. Note about triggers: • canonical way to disable triggers: ALTER TABLE DISABLE TRIGGER ALL; :( • pre 8.3 way: UPDATE pg_class.. • SET session_replication_role = ‘replica’; • Applies to just the current session • Thanks, Jan! en dp oi nt .c om
- 49. Dealing with failure • Troubleshooting • Backing up Bucardo itself • What happens when you lose the Bucardodb? en dp oi nt .c om
- 50. Bucardo hooks • LIVE DATA CHANGES! • pass in a hashref (ROW or ROWINFO) • Returned value - bitmapped (a/b or both!) • Affect the replication event, side effects • Example: cache invalidation en dp oi nt .c om
- 51. Bucardo Hooks • customcode & customcode_map • Conflict handlers (SWAP syncs) • Pure Perl • Exception - fix and try again • Pre and post trigger en dp oi nt .c om
- 52. Development • bucardo.org • mailing lists: http://bucardo.org/wiki/ Bucardo_mailing_list • Release early, often. • Please submit patches! bug reports! tests! en dp oi nt .c om
- 53. Questions? .com nt oi dp en
- 54. Thanks! http://www.slideshare.net/selenamarie/bucardo selena@endpoint.com @selenamarie on Twitter rss: http://www.chesnok.com/daily en dp oi nt .c om