MySQL Binary Log API Presentation - OSCON 2011
3 likes2,590 views
The document outlines an API for reading the MySQL binary log. It discusses how the API allows applications to connect to the MySQL server and read binary log events such as queries, row changes, and other events. It provides examples of using the API to iterate through events, read specific event data, and create custom event handlers to process the replication stream. The API is designed to be simple, extensible, and efficient for tasks like replicating to other systems, monitoring database changes, and point-in-time restore capabilities.
![First example
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) {
const char *url = “mysql://root@127.0.0.1:3360”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event.get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
12](https://image.slidesharecdn.com/osconbinlogapi-110730003415-phpapp02/85/MySQL-Binary-Log-API-Presentation-OSCON-2011-12-320.jpg)
![Create network transport
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) {
const char *url = “mysql://root@127.0.0.1:3360”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event.get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
13](https://image.slidesharecdn.com/osconbinlogapi-110730003415-phpapp02/85/MySQL-Binary-Log-API-Presentation-OSCON-2011-13-320.jpg)
![… or file transport
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) {
const char *url = “file:///tmp/binlog.0000001”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event.get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
14](https://image.slidesharecdn.com/osconbinlogapi-110730003415-phpapp02/85/MySQL-Binary-Log-API-Presentation-OSCON-2011-14-320.jpg)
![Connect the transport
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) {
const char *url = “file:///tmp/binlog.0000001”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event.get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
15](https://image.slidesharecdn.com/osconbinlogapi-110730003415-phpapp02/85/MySQL-Binary-Log-API-Presentation-OSCON-2011-15-320.jpg)
![Read events
#include <cstdlib>
#include <iostream>
#include <binlog_api.h> <Insert Picture Here>
int main(int argc, char *argv[]) { Get event
const char *url = “file:///tmp/binlog.0000001”;
Binary_log binlog(create_transport(url));
binlog.connect();
Binary_log_event *event;
while (true) {
int result = binlog.wait_for_next_event(&event);
if (result == ERR_EOF)
break;
cout << “ at “ << binlog.get_position()
<< “ event type “ << event>get_type_code()
<< endl;
}
return EXIT_SUCCESS;
}
17](https://image.slidesharecdn.com/osconbinlogapi-110730003415-phpapp02/85/MySQL-Binary-Log-API-Presentation-OSCON-2011-17-320.jpg)
![User-defined variables
class Save_handler : public Content_handler {
public:
<Insert Picture Here>
typedef std::map<std::string, std::string> Map;
Save_handler(Map &container) : m_var(container)
{ }
Binary_log_event *
process_event(User_var_event *event) {
m_var[event>name] = event>value;
return NULL;
}
private:
Map &m_var;
};
29](https://image.slidesharecdn.com/osconbinlogapi-110730003415-phpapp02/85/MySQL-Binary-Log-API-Presentation-OSCON-2011-29-320.jpg)
MySQL Binary Log API Presentation - OSCON 2011
- 2. <Insert Picture Here> An API for Reading the MySQL Binary Log Mats Kindahl Lars Thalmann Lead Software Engineer, MySQL Development Director, MySQL Replication & Utilities Replication, Backup & Connectors
- 3. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 3
- 4. Outline •Replication Architecture •Binary logs •Binary log event •Reading binary log – Connecting to server – Reading from files •Reading events – Queries – Reading rows (row-based replication) – Other events 4
- 5. Replication Architecture Master Slave(s) Clients Changes 5
- 6. Replication Architecture Master Slave(s) Session Databa Databa Databa se Session Client Dump Dump se Dump se Session I/O SQL I/O SQL I/O SQL Binary Log 6
- 7. Replication Architecture Master Slave(s) Session Databa Database Databa Session Client Dump Dump se Dump se Session I/O SQL I/O SQL I/O SQL Binary Log Relay Log 7
- 8. Replication to other systems Slave(s) Database Master I/O SQL Session Relay Log Session Client Dump Dump Dump Dump Data Session HBase Mining Full-text SOLR indexing Binary Log ? 8
- 9. Transforming events Subject of our presentation Se rv r e API Transformer SOLR le Fi “Change Data Capture” 9
- 10. Binlog API •Library to process replication events •API is ready for use •Goals: – Simple – Extensible – Efficient 10
- 11. <Insert Picture Here> Binlog API ● The replication listener How to capture events 11
- 12. First example #include <cstdlib> #include <iostream> #include <binlog_api.h> <Insert Picture Here> int main(int argc, char *argv[]) { const char *url = “mysql://root@127.0.0.1:3360”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event.get_type_code() << endl; } return EXIT_SUCCESS; } 12
- 13. Create network transport #include <cstdlib> #include <iostream> #include <binlog_api.h> <Insert Picture Here> int main(int argc, char *argv[]) { const char *url = “mysql://root@127.0.0.1:3360”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event.get_type_code() << endl; } return EXIT_SUCCESS; } 13
- 14. … or file transport #include <cstdlib> #include <iostream> #include <binlog_api.h> <Insert Picture Here> int main(int argc, char *argv[]) { const char *url = “file:///tmp/binlog.0000001”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event.get_type_code() << endl; } return EXIT_SUCCESS; } 14
- 15. Connect the transport #include <cstdlib> #include <iostream> #include <binlog_api.h> <Insert Picture Here> int main(int argc, char *argv[]) { const char *url = “file:///tmp/binlog.0000001”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event.get_type_code() << endl; } return EXIT_SUCCESS; } 15
- 16. Digression: set read position •Default: start at beginning <Insert Picture Here> •Set position explicitly: if (binlog.set_position(file, pos)) { /* Handle error */ } 16
- 17. Read events #include <cstdlib> #include <iostream> #include <binlog_api.h> <Insert Picture Here> int main(int argc, char *argv[]) { Get event const char *url = “file:///tmp/binlog.0000001”; Binary_log binlog(create_transport(url)); binlog.connect(); Binary_log_event *event; while (true) { int result = binlog.wait_for_next_event(&event); if (result == ERR_EOF) break; cout << “ at “ << binlog.get_position() << “ event type “ << event>get_type_code() << endl; } return EXIT_SUCCESS; } 17
- 18. Steps summary •Create a transport – create_transport •Connect to server – connect •Set position – set_position •Start event loop – wait_for_next_event 18
- 19. <Insert Picture Here> Binlog API ● The replication listener Reading information in events 19
- 20. Binlog Event Structure • Common header Common Header • Generic data • Fixed size • Post-header Post-header • Event-specific data • Fixed size • Variable part • Event-specific data Variable Part • Variable size 20
- 21. Reading the header •Read common header <Insert Picture Here> – header() •Access fields switch (event>header()>type_code) { case QUERY_EVENT: … case USER_VAR_EVENT: … case FORMAT_DESCRIPTION_EVENT: … } 21
- 22. Binlog Event Common Header type_code timestamp server_id • Data common to all events • Next Position 4 bytes – One-after-end of event • Timestamp – Statement start time • Flags – Binlog-in-use next_position – Thread-specific – Suppress “use” event_length – Artificial flags 19 Bytes – Relay-log event 22
- 23. Binlog Event Structure • Common header Common Header • Generic data • Fixed size • Post-header Post-header • Event-specific data • Fixed size • Variable part • Event-specific data Variable Part • Variable size 23
- 24. Query Event thread_id exec_time •Most common event Common Header •Used for statements •Statement logged literally query – … in almost all cases std::vector<uint8_t> variables ed ecod to be d error_code ne ed db_name ca se: Spe cial 24
- 25. Reading event data •Cast to correct event type <Insert Picture Here> •Access fields switch (event>header()>type_code) { case QUERY_EVENT: Query_event *qev = static_cast<Query_event*>(event); cout << qev>query << endl; break; case USER_VAR_EVENT: … case FORMAT_DESCRIPTION_EVENT: … } 25
- 26. Event-driven API <Insert Picture Here> 26
- 27. Event-driven API • Content handlers <Insert Picture Here> wait_for_next_event 27
- 29. User-defined variables class Save_handler : public Content_handler { public: <Insert Picture Here> typedef std::map<std::string, std::string> Map; Save_handler(Map &container) : m_var(container) { } Binary_log_event * process_event(User_var_event *event) { m_var[event>name] = event>value; return NULL; } private: Map &m_var; }; 29
- 30. Replace handler class Replace_vars : public Content_handler { Binary_log_event * process_event(Query_log_event *event) { /* Code to replace variables */ } p cp }; 2. s ic- ba le: mp e xa Full 30
- 31. <Insert Picture Here> Binlog API ● The replication listener Example two: How to capture live row changes 31
- 32. We'll cover this Row events in the binlog soon (trust me) Table map Header Write rows Transaction Write rows Write rows Table map Transaction Delete rows A bunch of rows Map table definition to table ID 32
- 34. Capturing row events • The *_ROWS_EVENT Defined in the table map event uint64_t table_id; uint16_t flags; uint64_t columns_len; uint32_t null_bits_len; vector<uint8_t> columns_before_image; vector<uint8_t> used_columns; vector<uint8_t> row; Raw row data 34
- 35. Reading rows •Wrap raw row data in Row_event_set •Iterate over rows using iterator Row_event_set rows(row_event, table_map_event); Row_event_set::iterator it= rows.begin(); You need to have captured this before! 35
- 36. Reading fields of a row •Row_of_fields to iterate fields of a row – Turns row into row of fields sequence Row_event_set rows(row_event, table_map_event); for (Row_event_set::iterator it = rows.begin() ; it != rows.end() ; ++it) table_delete(os.str(), Row_of_fields(*it)); 36
- 37. Reading fields of a row • Iterate over fields in Row_of_fields void table_delete (..., const Row_of_fields& fields) { Row_of_fields::iterator it= fields.begin(); for (int id = 0 ; it =! fields.end() ; ++it, ++id) { std::string str; Converter().to(str, *it); std::cout << id << "= " << str << std::endl; } } 37
- 38. Decoding a field • Iterate over fields in Row_of_fields void table_delete (..., const Row_of_fields& fields) { Row_of_fields::iterator it= fields.begin(); for (int id = 0 ; it =! fields.end() ; ++it, ++id) { std::string str; Converter().to(str, *it); std::cout << id << "= " << str << std::endl; } } 38
- 39. Summary – what's it for? •Replicate to other systems – Hbase, SOLR, etc. •Triggering on specific events – Call the DBA when tables are dropped? – Monitor objects in database •Browsing binary logs – Extract subset of changes (by table, by execution time, etc.) – Statistics •Component in building other solutions – Point-in-time restore – Sharding / Load-balancing 39
- 40. Summary – what we've covered •Reading events •Creating content handlers •Processing queries •Processing rows •Reading fields •… but there is a lot more 40
- 41. • Available at labs – http://labs.mysql.com/ • Source code available at launchpad – http://launchpad.net/mysql-replication-listener • MySQL High Availability Get it as free ebook: http://oreilly.com/go/ebookrequest Valid this week, mention event “MySQL Replication Update” 41