The powerful toolset of the go-mysql library

The powerful toolset of the go-mysql library
Daniël van Eeden
PingCAP
preFOSDEM MySQL Belgian Days 2025

Agenda
Introduction
Binlog
Server
Client

Who am I?
Daniël van Eeden.
Working for PingCAP on TiDB (MySQL Compatible database, written in Go). Long
time MySQL user.

Who am I?
Daniël van Eeden.
Working for PingCAP on TiDB (MySQL Compatible database, written in Go). Long
time MySQL user.
Contributor to:
▶ Wireshark (MySQL protocol decoding)
▶ TiDB
▶ MySQL
▶ DBD::mysql (Perl)
▶ . . .

Scope
Today I’ll demonstrate various usecases of go-mysql to you.

Project History
go-mysql was created by Siddon Tang in 2014 (VP of Engineering at PingCAP).
GitHub metrics: 177 contributors, 4.7k stars.
LICENSE: MIT, BSD-3-Clause

Usage
▶ Spirit (Online Schema Change Tool by Cash App, presented here last year)
▶ gh-ost (Online Schema Change Tool by GitHub)
▶ LoongCollector (Observability Data Collector by Alibaba)
▶ YDB Federated Query (by YDB Platform)
▶ TiDB DM (Data Migration Tool by PingCAP)
▶ . . . and 593 more repositories according to GitHub

Examples
Examples are made to fit on slides.

Examples
Error checking is left as an excercise to the reader.

Examples
Error checking is left as an excercise to the reader.
So is adding the right import statement
import (
"github.com/go-mysql-org/go-mysql/driver"
"github.com/go-mysql-org/go-mysql/client"
"github.com/go-mysql-org/go-mysql/mysql"
"github.com/go-mysql-org/go-mysql/server"
)

go-mysql
▶ go-mysql is native Go code, no C (cgo) is used.

replication
▶ The MySQL binlog records changes. (so no SELECT, etc)
▶ go-mysql can request the binlog stream from the server, just like a MySQL
replica.
▶ This requires the binlog to be enabled and requires the right permissions.
▶ Using binlog format=ROW is recommended.
▶ go-mysql can also read binlog files directly.

replication
func main() {
cfg := replication.BinlogSyncerConfig{
ServerID: 123,
Flavor: "mysql",
Host: "127.0.0.1",
Port: 3306,
User: "root",
Password: "",
}
syncer := replication.NewBinlogSyncer(cfg)
streamer, _ := syncer.StartSync(mysql.Position{"binlog.000003", 219532})
for {
ev, _ := streamer.GetEvent(context.Background())
if e, ok := ev.Event.(*replication.RowsEvent); ok {
for _, r := range e.Rows {
fmt.Printf("value of first column: %dn", r[0])
}
}
}
}

replication
[2025/01/22 17:24:19] [info] binlogsyncer.go:190 create BinlogSyncer with config {
,→ ServerID:123 Flavor:mysql Host:127.0.0.1 Port:3306 User:root Password:
,→ Localhost: Charset: SemiSyncEnabled:false RawModeEnabled:false TLSConfig:<
,→ nil> ParseTime:false TimestampStringLocation:UTC UseDecimal:false
,→ RecvBufferSize:0 HeartbeatPeriod:0s ReadTimeout:0s MaxReconnectAttempts:0
,→ DisableRetrySync:false VerifyChecksum:false DumpCommandFlag:0 Option:<nil>
,→ Logger:0xc0000a4300 Dialer:0x6ca180 RowsEventDecodeFunc:<nil>
,→ TableMapOptionalMetaDecodeFunc:<nil> DiscardGTIDSet:false EventCacheCount
,→ :10240 SynchronousEventHandler:<nil>}
[2025/01/22 17:24:19] [info] binlogsyncer.go:442 begin to sync binlog from
,→ position (binlog.000003, 219532)
[2025/01/22 17:24:19] [info] binlogsyncer.go:408 Connected to mysql 9.2.0 server
[2025/01/22 17:24:19] [info] binlogsyncer.go:869 rotate to (binlog.000003, 219532)
value of first column: 1

canal
type MyEventHandler struct {
canal.DummyEventHandler
}
func (h *MyEventHandler) OnRow(e *canal.RowsEvent) error {
for _, r := range e.Rows {
fmt.Printf("action=%s first_col=%#vn", e.Action, r[0])
}
return nil
}
func main() {
cfg := canal.NewDefaultConfig()
cfg.Addr = "127.0.0.1:3307"
cfg.User = "root"
cfg.Dump.TableDB = "test"
cfg.Dump.Tables = []string{"t"}
c, _ := canal.NewCanal(cfg)
c.SetEventHandler(&MyEventHandler{})
gtid, _ := mysql.ParseGTIDSet(mysql.MySQLFlavor,
"94e1b69b-d8df-11ef-85d2-c6bc6abebafa:1")
c.StartFromGTID(gtid)
}

replication
[2025/01/22 17:42:33] [info] dump.go:198 skip dump, use last binlog replication
,→ pos (, 0) or GTID set 94e1b69b-d8df-11ef-85d2-c6bc6abebafa:1
[2025/01/22 17:42:33] [info] binlogsyncer.go:462 begin to sync binlog from GTID
,→ set 94e1b69b-d8df-11ef-85d2-c6bc6abebafa:1
[2025/01/22 17:42:33] [info] sync.go:30 start sync binlog at GTID set 94e1b69b-
,→ d8df-11ef-85d2-c6bc6abebafa:1
[2025/01/22 17:42:33] [info] sync.go:59 received fake rotate event, next log name
,→ is binlog.000002
[2025/01/22 17:42:33] [info] sync.go:62 log name changed, the fake rotate event
,→ will be handled as a real rotate event
[2025/01/22 17:42:33] [info] sync.go:95 rotate binlog to (binlog.000002, 4)
[2025/01/22 17:43:18] [info] sync.go:248 table structure changed, clear table
,→ cache: test.t1
action=insert first_col="Hello (pre)FOSDEM!"
action=insert first_col="Have fun!"

server
type DemoHandler struct {
server.EmptyHandler
}
func (h DemoHandler) HandleQuery(query string) (*mysql.Result, error) {
if query == ‘SELECT 2+2‘ {
r, _ := mysql.BuildSimpleResultset(
[]string{"result"},
[][]interface{}{{"5"}}, false)
return mysql.NewResult(r), nil
}
return nil, nil
}
func main() {
l, _ := net.Listen("tcp", "127.0.0.1:4000")
c, _ := l.Accept()
conn, _ := server.NewConn(c, "root", "", DemoHandler{})
for {
if err := conn.HandleCommand(); err != nil {
panic(err)
}
}
}

replication
$ mysql -u root -h 127.0.0.1 -P 4000
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 10001
Server version: 8.0.11
Copyright (c) 2000, 2025, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ’help;’ or ’h’ for help. Type ’c’ to clear the current input statement.
mysql-8.0.11> SELECT 2+2;
+--------+
| result |
+--------+
| 5 |
+--------+
1 row in set (0.00 sec)

server
mysql> CREATE TABLE t1(id INT PRIMARY KEY);
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO t1 VALUES(1),(2),(3);
mysql> SELECT * FROM t1;
+------+
| id |
+------+
| 1 |
| 2 |
| 3 |
+------+
3 rows in set (0.00 sec)

server
mysql> SELECT sqlite_version();
+------------------+
| sqlite_version() |
+------------------+
| 3.46.0 |
+------------------+
mysql> PRAGMA table_list;
+--------+--------------------+-------+------+------+--------+
| schema | name | type | ncol | wr | strict |
+--------+--------------------+-------+------+------+--------+
| main | t1 | table | 1 | 0 | 0 |
| main | sqlite_schema | table | 5 | 0 | 0 |
| temp | sqlite_temp_schema | table | 5 | 0 | 0 |
+--------+--------------------+-------+------+------+--------+

server
mysql-8.0.11> SELECT * FROM generate_series(
-> ’2025-02-01 00:00:00’::TIMESTAMP,
-> ’2025-02-02 23:59:59’::TIMESTAMP,
-> ’12 hour’
-> );
+----------------------+
| generate_series |
+----------------------+
| 2025-02-01T00:00:00Z |
| 2025-02-01T12:00:00Z |
| 2025-02-02T00:00:00Z |
| 2025-02-02T12:00:00Z |
+----------------------+
mysql-8.0.11> SELECT VERSION();
+----------------------------------------------------------------------------------------
| version
|
+----------------------------------------------------------------------------------------
| PostgreSQL 17.2 (Debian 17.2-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debi
+----------------------------------------------------------------------------------------

server
Included:
▶ MySQL protocol
NOT Included:
▶ MySQL syntax (Both TiDB and Vitess have one in Go)
▶ Optimizer
▶ Storage Engine, etc

server
Usecases
▶ Security testing (clients, connectors, etc)
▶ Education
▶ Experimentation
▶ Making other data and/or services available

database/sql driver
go-mysql/driver is a pure-Go driver for database/sql.
Very similar to github.com/go-sql-driver/mysql.

database/sql driver
func main() {
db, err := sql.Open("mysql", "root@127.0.0.1:3307/test")
if err != nil {
panic(err)
}
defer db.Close()
var version string
db.QueryRow("SELECT VERSION()").Scan(&version)
fmt.Println(version)
}

database/sql driver
$ go run main.go
8.4.3

low-level client
func main() {
conn, err := client.Connect("127.0.0.1:3307", "root", "", "test")
if err != nil {
panic(err)
}
defer conn.Quit()
res, _ := conn.Execute("SELECT VERSION() AS ver")
defer res.Close()
version, _ := res.GetStringByName(0, "ver")
fmt.Println(version)
}

low-level client
$ go run main.go
8.4.3

low-level client
func main() {
conn, err := client.Connect("127.0.0.1:3307", "root", "", "test")
if err != nil {
panic(err)
}
defer conn.Quit()
fmt.Println(conn.GetServerVersion())
}

low-level client
First demo:
$ sudo tshark -i lo -p -f ’port 3307’ -Y mysql
Running as user "root" and group "root". This could be dangerous.
Capturing on ’Loopback: lo’
4 0.000494915 MySQL 143 Server Greeting proto=10 version=8.4.3
6 0.000550054 MySQL 210 Login Request user=root db=test
8 0.000648718 MySQL 77 Response OK
9 0.000667878 MySQL 94 Request Query SELECT VERSION() AS ver
10 0.000812981 MySQL 128 Response TABULAR Response
11 0.000856031 MySQL 71 Request Quit
6 packets captured
Second demo:
$ sudo tshark -i lo -p -f ’port 3307’ -Y mysql
Running as user "root" and group "root". This could be dangerous.
Capturing on ’Loopback: lo’
4 0.001084928 MySQL 143 Server Greeting proto=10 version=8.4.3
6 0.001273072 MySQL 210 Login Request user=root db=test
8 0.001595487 MySQL 77 Response OK
9 0.001772646 MySQL 71 Request Quit
4 packets captured

low-level client
Setup:
CREATE DATABASE IF NOT EXISTS test;
USE test;
DROP TABLE IF EXISTS t,mv;
CREATE TABLE t(
id int AUTO_INCREMENT PRIMARY KEY,
val int NOT NULL
);
CREATE TABLE mv (
id INT PRIMARY KEY,
val_avg decimal(8,4) NOT NULL
);
Now start demo-materialized-view

demo-materialized-view
mysql-8.4.3> INSERT INTO t(val) VALUES (1),(2);
Records: 2 Duplicates: 0 Warnings: 0

mysql-8.4.3> SELECT val_avg FROM mv;
+---------+
| val_avg |
+---------+
| 1.5000 |
+---------+

mysql-8.4.3> SELECT val_avg FROM mv;
+---------+
| val_avg |
+---------+
| 1.5000 |
+---------+
mysql-8.4.3> SELECT AVG(val) FROM t;
+----------+
| AVG(val) |
+----------+
| 1.5000 |
+----------+

mysql-8.4.3> EXPLAIN SELECT val_avg FROM mv;
+----+-------------+-------+------------+------+˜+------+----------+-------+
| id | select_type | table | partitions | type |˜| rows | filtered | Extra |
+----+-------------+-------+------------+------+˜+------+----------+-------+
| 1 | SIMPLE | mv | NULL | ALL |˜| 1 | 100.00 | NULL |
+----+-------------+-------+------------+------+˜+------+----------+-------+
1 row in set, 1 warning (0.00 sec)
mysql-8.4.3> EXPLAIN SELECT AVG(val) FROM t;
+----+-------------+-------+------------+------+˜+------+----------+-------+
| id | select_type | table | partitions | type |˜| rows | filtered | Extra |
+----+-------------+-------+------------+------+˜+------+----------+-------+
| 1 | SIMPLE | t | NULL | ALL |˜| 2 | 100.00 | NULL |
+----+-------------+-------+------------+------+˜+------+----------+-------+
1 row in set, 1 warning (0.00 sec)

2025/01/23 14:05:33 INFO initialized gtid=94e1b69b-d8df-11ef-85d2-c6bc6abebafa
,→ :1-14
[2025/01/23 14:05:33] [info] binlogsyncer.go:191 create BinlogSyncer with config
,→ ...
[2025/01/23 14:05:33] [info] binlogsyncer.go:463 begin to sync binlog from GTID
,→ set 94e1b69b-d8df-11ef-85d2-c6bc6abebafa:1-14
2025/01/23 14:05:50 INFO updating mv total=3 count=2 avg=1.5

What this demo does is:
1. Connect to MySQL and run SELECT COUNT(val), SUM(val) FROM t

2. Record the GTID position

3. Start to tail the binlog stream

4. Filter out row events for the t table.

5. Update the count and total based on the INSERT / UPDATE / DELETE
(once per transaction)

6. Calculate the average as total/count

6. Calculate the average as total/count
7. Write the average to the mv table with a REPLACE INTO statement (separate
connection).

Questions?
Thank you!
Daniel.van.Eeden@pingcap.com
https://github.com/go-mysql-org/go-mysql
https://github.com/dveeden/go-mysql-examples
Also speaking at:
▶ FOSDEM Cloud Native Databases devroom
- Distributed SQL Technologies: Raft, LSM Trees, Time, and More
- Saturday 17:35 UA2.114 (Baudoux)
- Co-speaker: Franck Pachot
▶ FOSDEM MySQL devroom
- MySQL Network Protocol: A walkthrough
- Sunday 14:15 H.1301 (Cornil)

The powerful toolset of the go-mysql library

More Related Content

Similar to The powerful toolset of the go-mysql library (20)

Recently uploaded (20)

The powerful toolset of the go-mysql library