MariaDB: in-depth (hands on training in Seoul)
18 likes9,263 views
MariaDB is a community-developed fork of MySQL that aims to be a drop-in replacement. It focuses on being compatible, stable with no regressions, and feature-enhanced compared to MySQL. The presentation covered MariaDB's architecture including connections, query caching, storage engines, and tools for administration and development like mysql, mysqldump, and EXPLAIN.
![mysql
mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or g.
Your MariaDB connection id is 11
Server version: 5.5.29-MariaDB Source distribution
Copyright (c) 2000, 2012, Oracle, Monty Program Ab and others.
Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)
23](https://image.slidesharecdn.com/mariadbin-depthtraining-130320191226-phpapp01/85/MariaDB-in-depth-hands-on-training-in-Seoul-23-320.jpg)
![Switching between
XtraDB & InnoDB
mysqld --ignore-builtin-innodb --plugin-
load=innodb=ha_innodb.so --plugin_dir=/usr/local/
mysql/lib/mysql/plugin
Or in my.cnf
[mysqld]
ignore-builtin-innodb
plugin-load=innodb=ha_innodb.so
plugin_dir=/usr/local/mysql/lib/mysql/plugin
79](https://image.slidesharecdn.com/mariadbin-depthtraining-130320191226-phpapp01/85/MariaDB-in-depth-hands-on-training-in-Seoul-79-320.jpg)
![Progress reporting
• ALTER TABLE & LOAD DATA INFILE
MariaDB [mail]> alter table mail engine = maria;
Stage: 1 of 2 'copy to tmp table' 17.55% of stage done
MariaDB [mail]> select id, user, db, command, state,
-> time_ms, progress from information_schema.processlist;
+---------+-------------------+-----------+----------+
| command | state | time_ms | progress |
+---------+-------------------+-----------+----------+
| Query | copy to tmp table | 23407.131 | 17.551 |
+---------+-------------------+-----------+----------+
1 row in set (0.47 sec)
92](https://image.slidesharecdn.com/mariadbin-depthtraining-130320191226-phpapp01/85/MariaDB-in-depth-hands-on-training-in-Seoul-92-320.jpg)
![New KILL syntax
• HARD | SOFT & USER USERNAME are
MariaDB-specific
• SOFT ensures things that may leave a table in an
inconsistent state aren’t interrupted (like REPAIR
or INDEX creation for MyISAM or Aria)
KILL [HARD | SOFT] [CONNECTION |
QUERY] [thread_id | USER user_name]
94](https://image.slidesharecdn.com/mariadbin-depthtraining-130320191226-phpapp01/85/MariaDB-in-depth-hands-on-training-in-Seoul-94-320.jpg)
MariaDB: in-depth (hands on training in Seoul)
- 1. MariaDB: in-depth Colin Charles, Monty Program Ab colin@mariadb.org | byte@bytebot.net http://montyprogram.com/ | http://mariadb.org/ http://bytebot.net/blog/ | @bytebot on Twitter Partner Training in Seoul, South Korea 19 March 2013 1
- 2. whoami • Work on MariaDB at Monty Program Ab • Previously from MySQL AB/Sun Microsystems • Have hacked on The Fedora Project and OpenOffice.org 2
- 3. who is he • Daniel Saito • APAC Sales Director, SkySQL • Former MySQL AB/Sun Microsystems/ Oracle • Former NTT Docomo 3
- 4. Agenda • 10.10am - we start • 12.30pm - 1.20pm - lunch • 1.30pm - we resume • 6pm - makgeoli time! (I’m not paying :P) • WiFi network is: 3Com (no password) • Please put mobiles on vibrate 4
- 5. We’re very social • Let’s use social media for the last time today (till the break!) - come say hi now! • facebook: fb.com/MariaDB.dbms • twitter: @MariaDB • g+: plus.google.com/+mariadb/ 5
- 6. What is MariaDB? • Community developed • MariaDB doesn’t depend on MySQL for • maria-captains: 50% MP, development - many features are developed 50% community including Sphinxsearch, Twitter, independently of MySQL SkySQL, Taobao, Facebook, Percona, Codership & • Backwards compatible with more MySQL • Feature enhanced • feature complete • better MySQL • replication supported for easy migration 6
- 7. Aims of MariaDB • Compatible, drop-in replacement to MySQL • your application shouldn’t care that its running MariaDB, easy upgrade (uninstall mysql, install mariadb, continue ops!) • Stable (bug-free) releases with no regressions • GPLv2 7
- 8. 3 years, many server releases • MariaDB 5.1, GA February 2010 • MariaDB 5.2, GA November 2010 • MariaDB 5.3, GA February 2012 • MariaDB 5.5, GA April 2012 • MariaDB Galera Cluster, GA March 2013 • MariaDB 10.0, alpha November 2012, release aim April 2013 8
- 9. 9
- 10. MariaDB 5.6? • SkySQL is fully committed to supporting MySQL 5.6 and all future GA releases of the server • For customers, SkySQL may create a branch of MySQL 5.6 with appropriate fixes & features from MariaDB if Oracle does not provide timely fixes 10
- 11. MariaDB 10.0 11
- 12. The MariaDB Ecosystem 12
- 13. Importance of understanding MariaDB (and MySQL) • Generics are inefficient • Since you have chosen MariaDB: • maximise its strengths • minimise its weaknesses 13
- 14. Sample databases to play with • http://dev.mysql.com/doc/index-other.html • sakila sample database, world database (used in MySQL training), menagerie database (used in book: Beginning MySQL), employees database (large dataset, comes with data - best to play with) 14
- 15. Picking hardware • Just use 64-bit hardware • VM’s are improving to use 64-bit OSes & software • Physical > virtual • Disk: battery backed storage, plan for RAID usage 15
- 16. Testing MySQL • Use MySQL Sandbox • http://mysqlsandbox.net/ • Express one-click MySQL installs • make_sandbox foo.tar.gz • Does not require root privileges • It is version aware: • http://code.openark.org/blog/mysql/cheating- mysqlsandbox-to-install-mariadb-10-0 16
- 17. Installation • Binaries (tarballs) are available at http://mariadb.org/ (source too) • Built, tested by MariaDB • Graphical installer & configuration for Microsoft Windows, with HeidiSQL GUI • Up-to-date monthly release schedule • RPM, DEB packages are provided, including YUM & APT repositories • Use the repository configuration tool • Inside Linux/*BSD distributions • Easy to install, basic defaults, may be older than upstream 17
- 18. Upgrades • Review changelogs carefully, even for minor versions • Make backups (using xtrabackup) • Don’t forget to run mysql_upgrade • by default with a distribution package • Replication can reduce downtime by upgrading the slave, promoting it, then upgrading the master 18
- 19. Packages (RPM example) • MariaDB-common • MariaDB-client • character sets • mysql client utilities • MariaDB-shared • MariaDB-test • latest libmysqlclient, some • test suite plugins, /etc/my.cnf.d/ • MariaDB-devel • MariaDB-compat • development headers enabling • compatible client libraries that you to build your own package replace mysql-libs • MariaDB-CassandraSE • MariaDB-server • plugin for CassandraSE • The actual server! 19
- 20. What is libmysqlclient? • Client libraries • Many applications are compiled against libmysqlclient • It is an Application Binary Interface (ABI) • When you write in Java & use Connector/J, Connector/J is compiled against libmysqlclient • Similarly with PHP & Connector/PHP 20
- 21. MySQL utilities • Why does MariaDB use all the same client utilities? • Because the aim is to be a 100% compatible drop-in replacement • There are very few non-MySQL utilities shipped: aria_chk, aria_dump_log, aria_ftdump, aria_pack, aria_read_log • xtstat (PBXT) is deprecated in MariaDB 5.5 • mytop as it adds features not-present upstream 21
- 22. mysql • Command line tool, just shell, works over SSH • Allows execution of statements, keeps track of settings, etc. 22
- 23. mysql mysql -uroot -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or g. Your MariaDB connection id is 11 Server version: 5.5.29-MariaDB Source distribution Copyright (c) 2000, 2012, Oracle, Monty Program Ab and others. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | +--------------------+ 3 rows in set (0.00 sec) 23
- 24. Finding help • How do you know what CLI options to use? • Use the man(ual) pages! • man mysql • man <command_name> usually works for any CLI command • HELP SELECT in mysql works too (help tables) - HELP <operator> 24
- 25. mysql_secure_insta llation • Now that you’ve tested MySQL works fine (by default, no root password), it’s time to ensure you secure the installation • Set root password • Remove anonymous users • Disallow remote root login • Remove test database • Reload privileges 25
- 26. mysqladmin • Manage user accounts, passwords, permissions • Display mysqld settings & status counters • Shutdown, create, drop databases PROCESSLIST SHOW shell> mysqladmin processlist shell> mysqladmin extended-status SHOW STATUS shell> mysqladmin -uroot -p create wordpress 26
- 27. mysqldump • Dump schema and/or data to .sql file, tab • Useful for backups, transferring data • Locks for consistency, so troublesome shell> mysqldump --all-databases > backup.sql shell> mysqldump --all-databases --single-transaction > backup.sql 27
- 28. Other tools • There are many other command line tools... • Front-ends • HeidiSQL • MySQL Workbench • Webyog’s SQLyog • Sequel Pro (OSX) 28
- 29. MariaDB Architecture 29
- 30. Making client connections • TCP/IP is available on all platforms • disable using --skip-networking • Socket files available on Unix • fastest communication path • MySQL connections are generally cheap • If you have many short running queries (typical web app), use MariaDB threadpool • Set connection limit max_connections=n in my.cnf 30
- 31. Connection Pool • mysqld’s main thread listens for connections • Each connection has a thread assigned to it • Threads can be: • created newly • reused from thread cache • User authentication processed based on hostname, username & password • Client specific buffers for session variables are allocated 31
- 32. Query Cache • Stores SELECT queries and their result sets • Subquery cache exists only in MariaDB • Frequently changed table data will cause queries to naturally be missed • Regularly replaced these days with memcached, et al. 32
- 33. SQL Parser • Lexical scanner & grammar rules • Parse SQL into tokens • Apply grammar rules to check statement validity • Construct a parse tree for Optimizer to use 33
- 34. Optimizer • Reads the parse tree and calculates the best query execution plan (QEP) to handle query • find indexes • determine JOIN order • eliminate unnecessary tables • etc. • The optimizer is smart, don’t try to force query plans 34
- 35. Pluggable Storage Engines • MariaDB can use many storage engines with different features • Installed/removed on fly with INSTALL/ UNINSTALL PLUGIN • Mixing & matching on same server, even on same query 35
- 36. Value proposition • No other database offers this capability • Unmatched flexibility + customisation potential • MEMORY engine for performance/routine lookup data • Right storage engine can improve performance in many applications • ARCHIVE compresses data, up to 80% • Partners & community benefit from this 36
- 37. What makes engines different? • Storage: how the data is stored on disk • Or in NDB (memory+disk), CassandraSE (access a Cassandra Cluster), SphinxSE (access the Sphinx daemon) • Indexes: improves search operations • Memory usage: improves data access for speed • Transactions: protects the integrity of your data (Atomic-Consistent-Isolated- Durable - ACID) • Locking level: MyISAM (table locks), InnoDB (row locks), old BDB (page locks) • Data types: Data types may be converted, MEMORY doesn’t support TEXT, etc. • Caching: InnoDB caches data & indexes, MyISAM caches indexes only (relying on OS disk cache for data) • Full-text search capability: MyISAM has this, InnoDB 5.6 just got this • GIS: MyISAM & Aria work (R-tree indexes exist), InnoDB 5.7 will probably have this 37
- 38. INFORMATION_SCHE MA • Holds metadata (data about the data) on all other databases & tables, exposed as regular tables • Generated on the fly • Has extensions in MariaDB as we expose more data than native MySQL 38
- 39. PERFORMANCE_SCH EMA • Allows for monitoring execution at a low level • This is a storage engine, monitoring server events (anything that takes time & can be instrumented) • Tables are views or temporary tables that use no on-disk storage • Such instrumentation isn’t free 39
- 40. MySQL 101 • q / ctrl+D works • no grants, no access • if statement is not terminated by • ERROR 1046 (3D000): No “;” (or uncommonly, g) and you hit database selected <-- only in Enter, you get another line for input databases can you make tables -> • CREATE TABLE foo (name • G - vertical column output VARCHAR(20), sex CHAR(1), birth DATE); • SHOW is a MySQL-ism • CREATE DATABASE foo; • SHOW TABLES; • USE foo; • DESCRIBE foo; • GRANT ALL on test.* TO ‘dbuser’@‘localhost’; 40
- 41. MySQL 101 • INSERT INTO foo • SELECT name FROM VALUES foo WHERE (‘fossasia’,’F’, sex=’f’; ‘2010-11-12’); • SELECT * FROM foo; • SELECT name FROM foo; 41
- 42. Use EXPLAIN 42
- 43. Ranges • SELECT * from room WHERE room_date between ‘2011-06-27’ AND ‘2011-07-02’; • must ensure that index is available on field being operated on by range operator (room_date) • too many records to return? Range optimisation won’t be used, and you get a index or full table scan 43
- 44. Scans & seeks • A seek, jumps into a random place (on disk/in memory) to fetch needed data • A scan, jumps to the start of data, and sequentially reads (on disk/in memory) until the end of needed data • Large amounts of data? Scan operations probably better than large amount of seeks 44
- 45. When do you get a full table scan • No WHERE condition • No index on any field with a WHERE condition • When range returns a large number of rows • SELECT * FROM 45
- 46. Primary keys • Always add one • When using InnoDB, use a small value (it is appended to every record in a secondary index) • Don’t add a PK? InnoDB adds one by default for you, and its a 6-byte integer 46
- 47. Schema/data types • MySQL has 9 to represent integers • Integer: TINYINT, SMALLINT, MEDIUMINT, INT, BIGINT • Floating point: FLOAT, DOUBLE • Fixed point: DECIMAL • Other: BIT 47
- 48. Bad schema design • INT(1) • this is NOT 1 digit. This is 4-bytes in size. A TINYINT is 1 byte • TINYINT UNSIGNED stores 0-255 values • BIT? If values are 0 or 1, can be used 48
- 49. Bad schema design • BIGINT AUTO_INCREMENT • INT UNSIGNED stores 4.3 billion values! • If you have a billion rows, think about PARTITIONing already 49
- 50. Not using UNSIGNED • Don’t want negative values? • Use UNSIGNED • Adds level of data integrity for cases where negative values might be present 50
- 51. CHAR(n) • This is what VARCHAR was built for • VARCHAR(n) works • Imagine storing simply 10 bytes in CHAR(128)? Wasted space 51
- 52. ONLINE ALTER • In MySQL 5.6 (and upcoming MariaDB 10) • 5.5 have external tools: • http://openarkkit.googlecode.com/svn/trunk/openarkkit/doc/ html/oak-online-alter-table.html • https://www.facebook.com/note.php?note_id=430801045932 • http://www.percona.com/doc/percona-toolkit/2.1/pt-online- schema-change.html • So, be very careful when you use ENUM columns • Probably only useful for sex (‘M’, ‘F’) or status (‘A’, ‘I’), since “Active” and “Inactive” is something you do in the presentation layer • Internally stored as 0 or 1 (very efficient) 52
- 53. Storing IPv4/6 addresses • INT UNSIGNED works • VARCHAR(15) results in an average of 12 bytes being used • ... versus 4 bytes via INT UNSIGNED • IPv6 have data types - INET6_ATON(expr) & INET6_NTOA(expr) • https://mariadb.atlassian.net/browse/ MDEV-274 53
- 54. Using NOT NULL • Can save a byte per column per row data, depending on the storage engine being used • Always define columns as NOT NULL unless there is a reason not to 54
- 55. Character sets • Default is UTF-8 • The web is UTF-8 • You can define this at the database, schema, table or even column level • MySQL internal buffers are fixed width • VARCHAR(255) utf8 is 765 bytes just to store 1 byte! • Efficient disk usage with VARCHAR, but internal memory usage isn’t 55
- 56. Tuning tips • 1,000 small queries is worse than 1 slow query! • Bytes? We save space because you want things in contiguous amounts of memory 56
- 57. Let’s go back to storage engines for a moment 57
- 58. Using different engines • Create a table with a specified engine • CREATE TABLE t1 (..) ENGINE=XtraDB; • Changing existing tables • ALTER TABLE t1 ENGINE=Aria; • SHOW ENGINES; 58
- 59. Transactional vs. non- transactional • Transaction-safe tables • Update fails? Changes (InnoDB) have advantages reverted over non-transaction safe tables (MyISAM): • Concurrency - tables w/ many update + • server crash? Automatic concurrent reads recovery, or a backup +transaction log • Disadvantages in today’s environments (transaction • ROLLBACK can be overhead = slower), more executed to ignore disk space requirements, changes more memory to perform updates don’t seem like they apply any longer 59
- 60. Indexes • Tree Indexes • Hash Indexes (MEMORY, NDB, InnoDB) • B-Trees • If table fits entirely in • B+Trees (InnoDB) memory, fastest way to perform queries is a hash • T-Trees (NDB) index • Red-black binary trees • InnoDB has an internal (MEMORY) adaptive hash index. InnoDB monitors index searches, • R-Trees (MyISAM for spatial and if it notices that it will benefit from a hash index, indexes) InnoDB automatically builds one. (5.1.24 and greater) 60
- 61. Does the storage engine really make a difference? MyISAM Inserts XtraDB Inserts ARCHIVE Inserts User Load Per Second Per Second Per Second 1 3,203.00 2,670.00 3,576.00 4 9,123.00 5,280.00 11,038.00 8 9,361.00 5,044.00 13,202.00 16 8,957.00 4,424.00 13,066.00 32 8,470.00 3,934.00 12,921.00 64 8,382.00 3,541.00 12,571.00 Using mysqlslap, against MariaDB 5.1.55, the ARCHIVE engine has 50% more INSERT throughput compared to MyISAM, and 255% more than XtraDB. 61
- 62. MyISAM • Pros? • no transactions • excellent INSERT • no foreign key performance support • small footprint • Typical uses • supports full-text • logging search (FTS) • auditing • Cons? • data warehousing 62
- 63. MyISAM II • In my.cnf, remember to • myisam_use_mmap set the key_buffer_size. enables MyISAM to use This is memory*0.40, as memory mapping (7-40% MyISAM uses the OS speed improvement) cache for tables • key_cache_segments = 1 • Using DRBD (which enables segmented key requires transactional caches in MariaDB - engine) but still want ~250% improvements, as FTS? Create a FTS it mitigates thread “slave” MyISAM table contention for key cache lock 63
- 64. InnoDB • Maintains its own buffer pool (does aggressive memory caching) • Uses tablespaces (several files on disk, raw disk support) • Typically used for OLTP operations 64
- 65. InnoDB: Tuning • innodb_file_per_table - • innodb_flush_log_at_trx split InnoDB data into a _commit=1 - logs file per table, rather than flushed to disk at each one contiguous file transaction commit. Guarantees ACID. • allows optimize table `table` to run • innodb_log_file_size - keep it high • innodb_buffer_pool_size (64-512MB), though recovery time increases =(memory*0.8) (largest=4GB) 65
- 66. InnoDB: Tuning II • SHOW ENGINE INNODB STATUS; • InnoDB supports both row & statement based replication • Assign a primary key, otherwise InnoDB assigns one for you. Also keep it small - its appended to every record in secondary index 66
- 67. ARCHIVE • Store large amounts of data without indexes, in small disk footprint • SELECT and INSERT operations only • Good for data audit use • Uses AZIO (zlib) compression 67
- 68. FederatedX • Create logical pointers to tables that exist on other MySQL servers; these can then be linked together to form one logical database • A federated table pointing to an InnoDB table on another server, will have transaction support (in 5.1) • Capabilities limited to underlying engine on remote server • CREATE TABLE t1 (...) ENGINE=FEDERATED CONNECTION='mysql://username:pwd@myhost:3306/db_name/ tbl_name • Can also be used for synchronous replication • Federated table on master server pointing to slave; triggers on master table to write all changes to remote table once applied to the master 68
- 69. Memory • Previously known as HEAP tables • In-memory engine • Hash index used by default, B-Tree available too • Typical uses? • Lookup tables • Session data • Temporary tables • Calculation tables • Cons? • Server dies, all rows are lost 69
- 70. Aria • Based off the 5.1 code • 1.0 – crash-safe MyISAM, with cacheable row format • 1.5 – concurrent INSERT/SELECT • Soon to be merged into 6.0, then... • 2.0: transactional + ACID compliance • 3.0: high concurrency, online backup • Goal: ACID compliant, MVCC transactional storage engine, based on MyISAM • Target? Data warehousing • Uses big log files (1GB by default) • 8K pages used by default (MyISAM uses 1K pages) • Has group commit (MariaDB 5.2) to speed up inserts 70
- 71. PBXT (deprecated) • MVCC, transactional, ACID compliant, foreign key support • row-level locking for updates, so maximum concurrency • immediate notification if client processes are deadlocked • write-once, as it uses a log-based architecture (write data to DB without first writing to transaction log) • support for BLOB streaming with Blob Streaming engine 71
- 72. Storage Engine API • http://dev.mysql.com/tech-resources/ articles/creating-new-storage-engine.html • SHOW PLUGINS; • https://kb.askmonty.org/v/extending-create- table • storage/example/ha_example.cc and storage/example/ha_example.h 72
- 73. Writing your own • Find the plugin path - show variables like “%plugins%”; +---------------+-----------------------------------+ | Variable_name | Value | +---------------+-----------------------------------+ | plugin_dir | /opt/maria/5.2.6/lib/mysql/plugin | +---------------+-----------------------------------+ • note that this is also where you store UDFs • Copy the relevant engine (eg. myengine.so) • INSTALL PLUGIN myengine SONAME 'myengine.so'; • Server registers plugin to mysql.plugin table, and now ENGINE=myengine will work 73
- 74. Things to think about • Backup is not engine-independent • MyISAM, InnoDB, PBXT • LVM/ZFS snapshots solve this • Different engines have different monitoring options • Mix and match; use summary tables 74
- 75. What others are using • Wordpress (blog): uses default engine, MyISAM is fine • MediaWiki (wiki): prefers InnoDB, except for “searchindex” table, which is MyISAM • http://svn.wikimedia.org/viewvc/mediawiki/trunk/ phase3/maintenance/tables.sql?view=markup • vBulletin (forum): MyISAM • SugarCRM (CRM): MyISAM (with conversion script to InnoDB provided) • Zimbra Collaboration Suite: InnoDB 75
- 76. Now, let’s focus on MariaDB features 76
- 77. We start with... • What came in 5.1, 5.2, 5.3, 5.5 (jump around appropriately) • What comes in 10.0 series • I won’t talk about deprecated features like OQGRAPH, PBXT in-depth, or even the old MariaDB 5.1 pool of threads (5.5 threadpool is better) 77
- 78. XtraDB • A more performant InnoDB designed to scale on modern hardware • Less checkpointing (smoother), less flushing to disk 78
- 79. Switching between XtraDB & InnoDB mysqld --ignore-builtin-innodb --plugin- load=innodb=ha_innodb.so --plugin_dir=/usr/local/ mysql/lib/mysql/plugin Or in my.cnf [mysqld] ignore-builtin-innodb plugin-load=innodb=ha_innodb.so plugin_dir=/usr/local/mysql/lib/mysql/plugin 79
- 80. MariaDB 5.5: an opensource threadpool • Modified from 5.1 (libevent • thread_handling=po based), great for CPU ol-of-threads bound loads and short running queries • https://kb.askmonty.org/en/ thread-pool-in-mariadb-55/ • Windows (threadpool), Linux (epoll), Solaris (event ports), FreeBSD/OSX (kevents) • No minimization of concurrent transactions with dynamic pool size 80
- 81. Better for DBAs: non- blocking client library • start operation, do work • https:// in thread, operation kb.askmonty.org/en/ processed, result travels about-non-blocking- back operation-in-the- client-library/ • use cases: multiple queries against single • fast node.js driver server (utilize more available: mariasql CPUs); queries against multiple servers (SHOW STATUS on • https:// kb.askmonty.org/en/ many machines) mariasql-for-nodejs/ 81
- 82. LIMIT ROWS EXAMINED • The purpose of this optimization is to • SELECT * from t1, t2 LIMIT 10 provide the means to ROWS EXAMINED terminate the execution 1000; of SELECTstatements which examine too many rows, and thus use • https://kb.askmonty.org/ en/limit-rows-examined/ too many resources. 82
- 83. SHOW STATUS • SHOW STATUS provides server status information. It is like mysqladmin extended-status. • SHOW STATUS LIKE ‘Key%’; • https://kb.askmonty.org/en/show-status/ • https://kb.askmonty.org/en/server-status- variables/ • MariaDB has opened_views, executed_triggers, executed_events, feature_* as new options 83
- 84. SQL Error Logging Plugin • Log errors sent to clients in a log file that can be analysed later. Log file can be rotated (recommended) • a MYSQL_AUDIT_PLUGIN install plugin SQL_ERROR_LOG soname 'sql_errlog.so'; 84
- 85. Replication: selective skipping • All changes that are logged as events in the binlog are replicated to all slaves • However, sometimes you want all to be logged to binlog but skipped replication to slaves • @@skip_replication (session only) • replicate_events_marked_for_skip = replicate|filter_on_slave| filter_on_master (dynamic) 85
- 86. Replication: dynamic variables • The variables replicate_do_*, replicate_ignore_*, and replicate_wild_* have been made dynamic, so they can be changed without requiring a server restart. • https://kb.askmonty.org/en/dynamic- replication-variables/ 86
- 87. Replication: Annotation of RBR events • MariaDB supports statement & row based replication (RBR) • In RBR, the binlog has no SQL statements, only events are logged (INSERT, DELETE, etc) • Option to include original SQL statement (default OFF) • https://kb.askmonty.org/en/ annotate_rows_log_event/ 87
- 88. Replication: binlog event checksums • Backport from MySQL 5.6 (in MariaDB 5.3+) • binlog_checksum option • Slaves perform checksums on events received & will stop if there is corruption • https://kb.askmonty.org/en/binlog-event- checksums/ 88
- 89. Replication: group commit in the binary log • sync_binlog=1, • http://kb.askmonty.org/en/group-commit- for-the-binary-log innodb_flush_log_at_trx_commi t=1 • SHOW STATUS LIKE 'binlog_%commits'; • https://www.facebook.com/note.php? note_id=10150261692455933 89
- 90. Replication: START TRANSACTION WITH CONSISTENT SNAPSHOT • Works with the binlog, possible to obtain the binlog position corresponding to a transactional snapshot of the database without blocking any other queries. • by-product of group commit in the binlog to view commit ordering • Used by the command mysqldump--single-transaction --master-data to do a fully non-blocking backup which can be used to provision a new slave • Works consistently between transactions involving more than one storage engine • https://kb.askmonty.org/en/enhancements-for-start-transaction- with-consistent/ 90
- 91. GIS precise • MySQL has OpenGIS SFS (Simple feature access, SQL access method) • Now, SQL with full geometry types • ST_ prefix • http://kb.askmonty.org/en/gis-features- in-533 91
- 92. Progress reporting • ALTER TABLE & LOAD DATA INFILE MariaDB [mail]> alter table mail engine = maria; Stage: 1 of 2 'copy to tmp table' 17.55% of stage done MariaDB [mail]> select id, user, db, command, state, -> time_ms, progress from information_schema.processlist; +---------+-------------------+-----------+----------+ | command | state | time_ms | progress | +---------+-------------------+-----------+----------+ | Query | copy to tmp table | 23407.131 | 17.551 | +---------+-------------------+-----------+----------+ 1 row in set (0.47 sec) 92
- 93. TIME_MS in I_S.PROCESSLIST • Extra column 'TIME_MS' has been added to the INFORMATION_SCHEMA.PROCESSLIST table • Units of milliseconds with microsecond precision (the unit and precision of the 'TIME' column is one second). 93
- 94. New KILL syntax • HARD | SOFT & USER USERNAME are MariaDB-specific • SOFT ensures things that may leave a table in an inconsistent state aren’t interrupted (like REPAIR or INDEX creation for MyISAM or Aria) KILL [HARD | SOFT] [CONNECTION | QUERY] [thread_id | USER user_name] 94
- 95. SphinxSE 95
- 96. The old days • Download MySQL, including sources • Download SphinxSE for compiling • Download Sphinx to compile with MySQL support • Documented: http://www.howtoforge.com/ sphinx-as-mysql-storage-engine-sphinxse 96
- 97. Today • Install sphinx from your distribution • Install MariaDB 5.5 from your distribution or from http://mariadb.org/ • Get started! 97
- 98. Getting started mysql> INSTALL PLUGIN sphinx SONAME 'ha_sphinx.so'; Query OK, 0 rows affected (0.01 sec) 98
- 99. Another engine appears 99
- 100. What is SphinxSE? • SphinxSE is just the storage engine that still depends on the Sphinx daemon • It doesn’t store any data itself • Its just a built-in client to allow MariaDB to talk to Sphinx searchd, run queries, obtain results • Indexing, searching is performed on Sphinx 100
- 101. Configure sphinx! • /usr/local/sphinx/sphinx.conf • Source (multiple, include mysql, with connection info) • Setup indexer (esp. if its on localhost) - mem_limit, max_iops, max_iosize • Setup searchd (where to listen to, query log, etc.) 101
- 102. Use case scenarios • Already have an existing application that makes use of full-text-search in MyISAM? Porting should be easier • Have a programming language without a native API for Sphinx? Surely there’s a connector for MariaDB ;-) 102
- 103. Use case scenarios • Results from Sphinx itself almost always require additional work involving MariaDB • Say to pull out text column that Sphinx index doesn’t store • JOIN with another table (using a different engine) 103
- 104. An example CREATE TABLE t1 ( id INTEGER UNSIGNED NOT NULL, weight INTEGER NOT NULL, query VARCHAR(3072) NOT NULL, group_id INTEGER, INDEX(query) ) ENGINE=SPHINX CONNECTION="sphinx://localhost:9312/ test"; SELECT * FROM t1 WHERE query='test it;mode=any'; 104
- 105. Sphinx search tables • 1st column: INTEGER UNSIGNED or BIGINT (document ID) • 2nd column: match weight • 3rd column: VARCHAR or TEXT (your query) • Query column needs indexing, no other column needs to be 105
- 106. What actually happens • SELECT passes a Sphinx query as the query column in the WHERE clause • searchd returns the results • SphinxSE translates and returns the results to MariaDB 106
- 107. SHOW ENGINE SPHINX STATUS • Per-query & per-word statistics that searchd returns are accessible via SHOW STATUS mysql> SHOW ENGINE SPHINX STATUS; +--------+-------+-------------------------------------------------+ | Type | Name | Status | +--------+-------+-------------------------------------------------+ | SPHINX | stats | total: 25, total found: 25, time: 126, words: 2 | | SPHINX | words | sphinx:591:1256 soft:11076:15945 | +--------+-------+-------------------------------------------------+ 2 rows in set (0.00 sec) 107
- 108. What queries are supported? • Most of the Sphinx API is exposed to SphinxSE • query, mode, sort, offset, limit, index, minid, maxid, weights, filter, !filter, range, !range, maxmatches, groupby, groupsort, indexweights, comment, select • Sphinx search modes can also be supported via _sph attributes • obtain value of @groupby? use ‘_sph_groupby’ 108
- 109. Efficiency • Allow Sphinx to perform sorting, filtering, and slicing of result set • ... as opposed to using WHERE, ORDER BY, LIMIT clauses on MariaDB • Why? • Sphinx optimises and performs better on these tasks • Less data packed by searchd, and transferred and unpacked by SphinxSE 109
- 110. JOINs • Perform JOINs on a SphinxSE search table using tables from other engines SELECT content, date_added FROM test.documents docs -> JOIN t1 ON (docs.id=t1.id) -> WHERE query="one document;mode=any"; +-------------------------------------+---------------------+ | content | docdate | +-------------------------------------+---------------------+ | this is my test document number two | 2006-06-17 14:04:28 | | this is my test document number one | 2006-06-17 14:04:28 | +-------------------------------------+---------------------+ 2 rows in set (0.00 sec) 110
- 111. User statistics • Understand server activity better, identify database loads http://kb.askmonty.org/v/ user-statistics • Must be enabled first • /usr/local/Cellar/mariadb/5.2.7/bin/ mysqld_safe --datadir=/usr/local/var/mysql --userstat=1 111
- 112. userstats 112
- 113. Table Elimination • Resolve a query without accessing some tables query refers to • Great for querying highly normalised data • Basis of “anchor modelling” • http://www.anchormodeling.com/ • SQL Server 2005/2008, Oracle 11g have it 113
- 114. Virtual columns • Columns that are an expression that are calculated on retrieval • PERSISTENT or VIRTUAL • Similar to MS SQL or Oracle • https://kb.askmonty.org/en/virtual-columns/ 114
- 115. MyISAM segmented key caches • Mitigates thread contention for key cache lock, with notable performance improvements • Key caches divided into different segments, allowing for better key cache concurrency • 1-64 segments 115
- 116. Optimizer enchancements • Join additions • block nested loop joins for outer-joins, block hash joins, Batched Key Access (BKA) • Optimization for derived tables & views • mergeable derived tables processed like VIEWs + optimizer creates indexes over materialized derived tables • Disk access optimization • Index Condition Pushdown (ICP), Multi-Range Read (MRR) 116
- 117. Subquery optimizations • Semi-join optimization, • Goodbye rewriting as materialization for non- JOINs or separate correlated IN queries, queries subquery cache DBT-3, 60M rows, 29GB XtraDB 117
- 118. 118
- 119. EXPLAIN, DBT-3 • INSTANT EXPLAIN • EXPLAIN Analyzer - https://mariadb.org/ explain_analyzer/ analyze/ • Optimizer feature comparison matrix: https://kb.askmonty.org/ en/optimizer-feature- comparison-matrix/ 119
- 120. Extended keys • Default is extended_keys=off • Extended Keys, introduced in MariaDB 5.5, is an optimization which makes use of existing components of InnoDB/XtraDB keys to generate more efficient execution plans. Using these components in many cases allows the server to generate execution plans which employ index-only look-ups. 120
- 121. NoSQL • Comes with HandlerSocket • direct access to XtraDB/InnoDB for CRUD operations • INSTALL PLUGIN handlersocket SONAME 'handlersocket.so'; • Dynamic columns • create columns with dynamic content • basically a blob with handling functions 121
- 122. Pluggable authentication • MariaDB & MySQL now uses password authentication via pluggable auth • Unix sockets • PAM • Active Directory (MySQL) • LDAP (interested?) • Kerberos (interested?) 122
- 123. Windows performance OLTP read-only OLTP write-only 123
- 124. MariaDB 10.0 124
- 125. Why MariaDB 10.0? • The 5.5 merge took about a year (!) • We (MariaDB-5.5) have over 1.5 million lines of extra code with a ~61MB diff • We didn’t want to repeat this for 5.6 • Also, MySQL 5.6 has a lot of re-factoring, thus loosing commit history 125
- 126. In a nutshell • Built on MariaDB 5.5 • Backported features from MySQL 5.6 • New features 126
- 127. What about tools? • SELECT VERSION() will • Use 9.0 for a name return 10.0.1-MariaDB • Lie to clients (no) • Oops, we found a bug in MySQL: https:// mariadb.atlassian.net/ • Disallow replication (no) browse/MDEV-4088 & http://bugs.mysql.com/ bug.php?id=68187 • Use handshake packet 5.5.30-mysql-10.0.2- MariaDB without • Still deciding: affecting VERSION() / @@global.version 127
- 128. What about tools II? • Tools really should recognise MariaDB version as there are already many new features that MySQL doesn’t have • eg. HeidiSQL supports virtual columns (http://www.heidisql.com/forum.php? t=8671) 128
- 129. Versions Date Version Status 12 Nov 10.0.0 Alpha 2012 6 Feb 10.0.1 Alpha 2013 22 Mar 10.0.2 Beta 2013 10.0.3 GA 129
- 130. Backported features (i.e. these are from MySQL 5.6) 130
- 131. InnoDB & XtraDB • MariaDB 10.0 ships InnoDB from MySQL 5.6 • MariaDB 10.0.2/3 will probably have InnoDB Full Text Search enabled • Percona XtraDB ~ mid-2013 • no performance improvements expected, just functionality & features • bitmap changed page tracking so xtrabackup can do incremental backups without scanning all InnoDB files • XtraDB when GA can be used again 131
- 132. More from MySQL 5.6 • PERFORMANCE_SCHEMA • InnoDB read-only transactions (TRANSACTION READ ONLY) • Optimizer: • EXISTS-TO-IN optimization • ORDER BY...LIMIT optimization (show only few rows of a result set) • CURRENT_TIMESTAMP as DEFAULT for DATETIME columns (this is a re-implementation in MariaDB) 132
- 133. Only in MariaDB 10.0 the new stuff! 133
- 134. Multi-source replication • Work from Taobao • Many users partition data across many masters... now you can replicate many masters to a single slave • Great for analytical queries, complete backups, etc. • https://kb.askmonty.org/en/multi-source- replication/ 134
- 135. Only in 10.0 • SHOW EXPLAIN for • Per-thread memory usage <thread_id> (https:// (Taobao) kb.askmonty.org/en/show- explain/) gets the query plan of a running statement • I_S.PROCESSLIST has MEMORY_USAGE & EXAMINED_ROWS • Faster ALTER TABLE with unique keys for Aria & MyISAM • SHOW STATUS has memory usage too • Segmented MyISAM keycaches (up to 64) since MariaDB 5.2 exist too 135
- 136. CassandraSE 136
- 137. CassandraSE • Integration with NoSQL/Big Data DB, Apache Cassandra cluster, seen as a storage engine to MariaDB • Combine (join) data between Cassandra & MariaDB • Write to Cassandra from SQL (SELECT, INSERT, UPDATE, DELETE) • CQL is great, but the goal is for you to just work with SQL, not switch between CQL & SQL • Data is mapped: rowkey, static columns, dynamic columns • super columns aren’t supported • No 1-1 direct map for data types (ref: https://kb.askmonty.org/en/ cassandra-storage-engine/) 137
- 138. LevelDB Storage Engine • LevelDB is a key-value store, used even in the Chrome web browser • LevelDBSE supports single-statement transactions, secondary indexes, crash- proof slave replication, hot backups & more 138
- 139. Engine-independent persistent statistics • InnoDB has persistent statistics in MySQL 5.6; we have an engine-independent version • These statistics aren’t limited by the SE API, and are used by query optimizer to choose best execution plan for each statement • Statistics collected for non-indexed columns too (unlike InnoDB’s) 139
- 140. Dynamic columns • Store a different set of columns for every row in the table (kinda NoSQL-like) • Database interoperability is now a major feature; CassandraSE makes uses of this • Can now request a row in JSON format • And yes, you can name columns finally ;) 140
- 141. What about MySQL 5.6? • We love the fact that many features we’ve worked on for a long time are now in 5.6 • Optimizer enhancements • Microseconds • Binary log annotations • Binary log group commit (10.0 has a newer faster version now) • Precise GIS • Threadpool 141
- 142. What are we missing from 5.6 currently? • EXPLAIN output in JSON • Online operations (ALTER TABLE) • InnoDB FTS • Global transaction IDs • InnoDB memcached interface • Many will be ready in final 10.0 release 142
- 143. Support • Five years from every release 143
- 144. Benchmarks • “Lies, damned lies, and statistics” - Mark Twain • http://blog.mariadb.org/sysbench-oltp- mysql-5-6-vs-mariadb-10-0/ • http://dimitrik.free.fr/blog/archives/2013/02/ mysql-performance-mysql-56-vs-mysql-55- vs-mariadb-55.html • Yes, we’ve gotten Oracle to notice MariaDB :-) 144
- 145. Continued commitments • Security • Since about a year now, we’re the go-to people for security - good track record • We don’t like regressions • http://www.skysql.com/blogs/hartmut/nasty-innodb-regression- mysql-5525 • http://www.skysql.com/blogs/kolbe/heads-no-more-query-cache- partitioned-tables-mysql-5523 • We care about backward compatibility & introduce features carefully • XtraDB innodb_adaptive_checkpoint=none|reflex|estimate| keep_average (no more reflex...) 145
- 146. What’s next • MariaDB Galera Cluster • Automated failover with MHA & MariaDB 146
- 147. What we didn’t discuss • Configuring MariaDB (MySQL-generics) • use a configurator tool • Proper backups • NDB Cluster • MySQL features like VIEWs, PARTITIONING, event scheduler, etc. Go buy a MySQL training class • Monitoring from SkySQL! :-) • Optimization 147
- 148. Q&A / Thanks colin@mariadb.org / byte@bytebot.net @bytebot on twitter | http://bytebot.net/blog/ 148