SlideShare a Scribd company logo
MySQL stuff
About me
• Before REA I worked for a hosting company for 4
years
• Learnt a lot about MySQL during this period
• I started as an OPS engineer at REA and did some
database stuff here
• I havent done much MySQL for two years
!
• In this period we’ve had
• MySQL 5.6 come out
• RDS MySQL improvements
!
!
Today
• High level overview of MySQL
• Look inside MySQL server
• Talk through how InnoDB performs queries
• Schema changes and online schema changes
!
!
Background
• Second biggest relational database in the world
• Anyone know what the biggest is?
!
• Used by many of the big web shops
• Facebook for all feed data
• Etsy for user & data
• Twitter for tweet persistence
• …
!
• Sun bought MySQL AB in 2008, people freaked out
• Oracle bought Sun in 2010, people freaked out
• Big improvements over the past few years (5.5 / 5.6)
!
How it works
mysqld
mysql client tcp:3306
filesocket
ServerClient
libmysql
mysql lib
ORM
host$ mysql
host$ mysqld_safe!
host$ mysqld
Big features
• Pluggable storage engines is massive flexibility
• You can use a different storage engine per table
!
!
• Common storage engines
• InnoDB - ACID compliant store
• MyISAM - one of the original storage engines -
avoid
• Lots and lots of others
!
Inside mysqld
mysqld
storage engine(s)
MyISAM / InnoDB / Memory / etc
tcp:3306
filesocket
Getting the query
and responding
bits and
bytes stuff
API
Inside mysqld
mysqld
storage engine(s)
MyISAM / InnoDB / Memory / etc
tcp:3306
filesocket
API
Inside mysqld
mysqld
storage engine(s)
MyISAM / InnoDB / Memory / etc
tcp:3306
filesocket
Parse query and
convert to SE
API calls.
API
Inside mysqld
mysqld
Storage engines(s)
tcp:3306
filesocket
API
query cache
binary log
join / sort buffers
Binary Log for replication
Buffers: Kinda how much
batching to storage API
to save cost
The bad parts - binary log
mysqld
tcp:3306
filesocket
query cache
binary log
join / sort buffers
Because you can mix and
match storage engines,
replicating state has to be
done at the lowest
common denominator
Innodb
Table 1
MyISAM
Table 2
The bad parts - binary log
mysqld
Innodb
Table 1
tcp:3306
filesocket
query cache
binary log
join / sort buffers
Highly concurrent InnoDB
needs its queries,
serialized down
to replicate to slave
MyISAM
Table 2
Compare to other DBs
DB server
tcp:x
filesocket?
performance
knobs
transaction
log
The bad parts - query cache
mysqld
Innodb
Table 1
tcp:3306
filesocket
query cache
binary log
join / sort buffers
Can only work on an
entire table level &
uses LRU
MyISAM
Table 2
The bad parts - query cache
mysqld
Innodb
Table 1
tcp:3306
filesocket
query cache
binary log
join / sort buffers
All ‘update/insert/delete’
blocks all transactions
to ensure correct state.
It actually slows !
things down
MyISAM
Table 2
Inside mysqld
mysqld
storage engine(s)
MyISAM / InnoDB / Memory / etc
tcp:3306
filesocket
API
Where 10%* of !
the work is done
Where 90% !
of the work !
is done
*Finger in the air calculation
Summary
• Turn off query cache! It is hurting you!
• Binary log is a necessary evil for replication
• Dont play with the performance knobs (sort buffer /
join buffer / etc)
• Turning the 10% - Focus on the 90%
• Use InnoDB for all the tables (unless they’re system
tables)
!
!
• So…lets look at the 90%
!
!
InnoDB
mysqld
InnoDB
tcp:3306
filesocket
API
Unknown gems
mysqld
InnoDB
tcp:3306
filesocket
API
handlersocket/
memcache
Innodb
Unknown gems
mysqld
InnoDB
tcp:3306
filesocket
API
handlersocket/
memcache
Innodb
Memcache API
that talks directly
to InnoDB
!
Removes the SQL
layer and is just K/V
!
Google: handlersocket
memcache InnoDB
Key parts of InnoDB
Innodb
API
Key parts of InnoDB
Table space
(whats on disk)
redo log
Innodb
bufferpool
memory!
ib_log1!
ib_log2!
sometable.ibd!
ib_data!
Key parts of InnoDB
Table space
redo log
bufferpool
• Each block is a page
• Just think of it as a row with some extra stuff like
version number
• InnoDB is ACID meaning each connection has to
have its own view of the world!
• Extra metadata at the top about where everything is
Key parts of InnoDB
Table space
redo log
bufferpool
• Black is the base metadata
What is bufferpool
Table space
redo log
bufferpool
• In memory version of hot tablespace pages
What is redo log
Table space
redo log
bufferpool
• An append only log file of changes
What is table space
Table space
redo log
bufferpool
• The raw files - but it may not be always up-to-date
Lets run some queries
Table space
redo log
bufferpool
• First - lets fill up our database with data
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
mysqld parses the query and sends !
API calls to InnoDB
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
Is red in bufferpool?
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
Load red into buffer pool
perform random !
read disk activity!
At this point, the time to respond!
is dependent on disk speed!
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
Respond to API calls to mysqld
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
Parse results !
second pass sort if necessary !
return to client
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red; #again
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
mysqld parses the query and sends !
API calls to InnoDB
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
Is red in bufferpool? - yes
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
Respond to API calls to mysqld
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from red;
Parse results !
second pass sort if necessary !
return to client
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from brown;
What if bufferpool is full?
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from blue;
Assume metadata knows page requirements!
LRU on the bufferpool to find space and evict
Lets run some queries
Table space
redo log
bufferpool
• mysql> select * from blue;
!
Load in blue as before and return
Lets run some queries
Table space
redo log
bufferpool
• mysql> update blue set A=B where ID=1;
!
Is page in bufferpool? yes
Lets run some queries
Table space
redo log
bufferpool
• mysql> update blue set A=B where ID=1;
!
Update the page in bufferpool and increment !
page version number
Lets run some queries
Table space
redo log
bufferpool
• mysql> update blue set A=B where ID=1;
!
Write the new page to the redo log!
what tablespace is up to!
Lets run some queries
Table space
redo log
bufferpool
• mysql> update blue set A=B where ID=1;
!
Update metadata telling it that this is the !
new state of the page!
perform sequential !
write disk activity to redo log!
This is pretty fast!
what tablespace is up to!
Lets run some queries
Table space
redo log
bufferpool
• mysql> update blue set A=B where ID=1;
!
Send the OK back to mysqld!
what tablespace is up to!
Lets run some queries
Table space
redo log
bufferpool
• mysql> update blue set A=B where ID=1;
!
At sometime in the near future, do a semi sequential !
parse changes and update the tablespace
what tablespace is up to!
semi sequential write!
Not that bad !
!
Lets run some queries
Table space
redo log
bufferpool
• mysql> insert into yellow values
(1,2,3,4);
!
Is space in bufferpool?!
Lets run some queries
Table space
redo log
bufferpool
• mysql> insert into yellow values
(1,2,3,4);
!
Is adjacent pages in bufferpool?!
Random read !
(but not too big)
Lets run some queries
Table space
redo log
bufferpool
• mysql> insert into yellow values
(1,2,3,4);
!
Send OK back to client!
Key take aways
Table space
redo log
bufferpool
• Cold databases are slow!
• bufferpool is empty and everything requires random disk
reads
• The bigger the buffer pool you have, the quicker you will be!
(NUMBER 1 factor on performance!)!
• Redo log is the second slowest part
• we can tune this
innodb_flush_logs_at_trx_commit=1
innodb_buffer_pool_size= ~80% memory!
Schema changes
• Everyone hates them
• They are not transactional
• They are blocking (well most)
• How do they work?
!
Remember this?
mysqld
Storage engines(s)
tcp:3306
filesocket
API
query cache
binary log
join / sort buffers
Lets add a few more
Storage engines(s)
tcp:3306
filesocket
API
table meta data
binary log
Table triggers
functions / procedures
mysqld
tcp:3306
filesocket
mytbl
Schema changes
mytbl meta
mytbl
mysql> CREATE TABLE mytbl 

(a INT, b CHAR (20), INDEX (a)) ENGINE=InnoDB;
Schema changes
mytbl
Schema changes
mysql> ALTER TABLE mytbl 

ADD COLUMN age INT;
mytbl
Schema changes
mysql> ALTER TABLE mytbl 

ADD COLUMN age INT;
!
Wait for any open transactions to close and then !
lock access to the table!
block
mytbl
Schema changes
mysql> ALTER TABLE mytbl 

ADD COLUMN age INT;
!
Create a new ‘hidden’ table with the new schema!
block
.mytbl
mytbl
Schema changes
mysql> ALTER TABLE mytbl 

ADD COLUMN age INT;
!
Do an internal ‘mysqldump’ / ‘mysqlrestore’!
block
.mytbl
mytbl
Schema changes
mysql> ALTER TABLE mytbl 

ADD COLUMN age INT;
!
Do a switch-a-roo!
block
.mytbl
Schema changes
mysql> ALTER TABLE mytbl 

ADD COLUMN age INT;
!
Unblock and send OK back to client!
mytbl
Online schema changes
• Some changes can be done without ‘locking’
• Column additions
• Index additions
!
• Any removal cannot be done online
• Unless you do it yourself
!
mysqld
mytbl
Online schema changes 5.1/5.5
mytbl meta
mysqld
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER…
pt-online-schema-change
soundclouds lhm
etc
mysqld
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER… .mytbl meta
.mytbl
!
Creates another
table with the
updated
schema in the
client!
mysqld
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER… .mytbl meta
.mytbl
!
Creates a set of
triggers on the
main table!
triggers
mysqld
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER… .mytbl meta
.mytbl
!
selects the
entire table (in
batches) and
inserts into the
new table!
!
!
This can take a
long time!
triggers
mysqld
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER… .mytbl meta
.mytbl
triggers
client
mysql> select
!
Whilst the new
table is being
built, selects
still go to the
old table!
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER… .mytbl meta
.mytbl
triggers
client
mysql> UPDATE
!
Updates cause
the trigger to
fire which
updates the
new table being
built. (Inserts
the row if its
still hasn't been
copied)!
!
Deletes happen
the same way!
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER… .mytbl meta
.mytbl
triggers
client
mysql> INSERT
!
Inserts also
happen the
same way
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER… .mytbl meta
.mytbl
triggers
!
Eventually the
table
population
finishes and a
‘RENAME table’
action is
performed
(which is quick)
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER… .mytbl meta
.mytbl
triggers
!
Eventually the
table
population
finishes. This is
the only
‘blocking’ time
block block
mytbl
Online schema changes 5.1/5.5
mytbl meta
client
mysql> ALTER… !
The schema
change tool
finishes
Online schema changes in 5.6
• All that ‘switch-a-roo’ logic is now within mysqld itself
!
Summary about schema changes
• You ‘double your IO’ when you’re online schema
changes!
• Binary log is outside of this - so watch out for your
slaves table locking if you don't use the right flags
!
!

More Related Content

PPTX
HBaseConEast2016: Coprocessors – Uses, Abuses and Solutions
PPTX
HBaseConEast2016: How yarn timeline service v.2 unlocks 360 degree platform i...
PDF
Modern Linux Performance Tools for Application Troubleshooting
PDF
Cascading - A Java Developer’s Companion to the Hadoop World
PPTX
Overview of Cascading 3.0 on Apache Flink
PDF
Presto anatomy
PDF
GNW01: In-Memory Processing for Databases
PPTX
LLAP: Locality is dead (in the cloud)
HBaseConEast2016: Coprocessors – Uses, Abuses and Solutions
HBaseConEast2016: How yarn timeline service v.2 unlocks 360 degree platform i...
Modern Linux Performance Tools for Application Troubleshooting
Cascading - A Java Developer’s Companion to the Hadoop World
Overview of Cascading 3.0 on Apache Flink
Presto anatomy
GNW01: In-Memory Processing for Databases
LLAP: Locality is dead (in the cloud)

What's hot (20)

PDF
Empowering developers to deploy their own data stores
PDF
Lessons PostgreSQL learned from commercial databases, and didn’t
PDF
High Concurrency Architecture and Laravel Performance Tuning
PPTX
Low Level CPU Performance Profiling Examples
PPTX
HBaseConEast2016: Splice machine open source rdbms
PDF
Online Schema Changes for Maximizing Uptime
PDF
Apache Flink vs Apache Spark - Reproducible experiments on cloud.
PDF
Kafka Summit SF 2017 - Streaming Processing in Python – 10 ways to avoid summ...
PDF
Apache Big Data EU 2015 - Phoenix
PDF
MySQL5.7 Innodb_enhance_parti_20160317
PDF
Training Slides: Basics 103: The Power of Tungsten Connector / Proxy
PDF
Presto changes
PDF
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
PDF
0.5mln packets per second with Erlang
PDF
From Zero to Hero with Kafka Connect
PDF
Presto - Analytical Database. Overview and use cases.
PPTX
January 2015 HUG: Apache Flink: Fast and reliable large-scale data processing
PDF
Postgres vision 2018: The Promise of zheap
 
PDF
Building a High-Performance Database with Scala, Akka, and Spark
PDF
Postgres Vision 2018: WAL: Everything You Want to Know
 
Empowering developers to deploy their own data stores
Lessons PostgreSQL learned from commercial databases, and didn’t
High Concurrency Architecture and Laravel Performance Tuning
Low Level CPU Performance Profiling Examples
HBaseConEast2016: Splice machine open source rdbms
Online Schema Changes for Maximizing Uptime
Apache Flink vs Apache Spark - Reproducible experiments on cloud.
Kafka Summit SF 2017 - Streaming Processing in Python – 10 ways to avoid summ...
Apache Big Data EU 2015 - Phoenix
MySQL5.7 Innodb_enhance_parti_20160317
Training Slides: Basics 103: The Power of Tungsten Connector / Proxy
Presto changes
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
0.5mln packets per second with Erlang
From Zero to Hero with Kafka Connect
Presto - Analytical Database. Overview and use cases.
January 2015 HUG: Apache Flink: Fast and reliable large-scale data processing
Postgres vision 2018: The Promise of zheap
 
Building a High-Performance Database with Scala, Akka, and Spark
Postgres Vision 2018: WAL: Everything You Want to Know
 
Ad

Viewers also liked (20)

DOCX
Operation research complete
PPTX
Happy halloween!
PDF
Mobile Computing IEEE 2014 Projects
PDF
Imperious group presentation
PPTX
DOC
Financial management
PPTX
devopsdays Rome
PDF
2014 Investor Day
PPT
Cadmun economics proyect
PDF
Inhibition, kinetic and thermodynamic effects of new Azo derivatives on iron ...
PPTX
Yhteisen talouden talkoot
PDF
صفات قلب الراعي
PPTX
Cory mosley what matters most
PPTX
Hebrew Kings
PPTX
An Overview of VIEW
DOC
026 30 03-2014 am الكلمة الخامسة - أنا عطشان
PPTX
Farewell Sermon
PDF
Microsoft Power Point Brett Campbellv2 [Compatibility Mode]
Operation research complete
Happy halloween!
Mobile Computing IEEE 2014 Projects
Imperious group presentation
Financial management
devopsdays Rome
2014 Investor Day
Cadmun economics proyect
Inhibition, kinetic and thermodynamic effects of new Azo derivatives on iron ...
Yhteisen talouden talkoot
صفات قلب الراعي
Cory mosley what matters most
Hebrew Kings
An Overview of VIEW
026 30 03-2014 am الكلمة الخامسة - أنا عطشان
Farewell Sermon
Microsoft Power Point Brett Campbellv2 [Compatibility Mode]
Ad

Similar to Howmysqlworks (20)

PPTX
20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02
PDF
The InnoDB Storage Engine for MySQL
PPTX
Rubyslava + PyVo #48
PPTX
Introduction to TokuDB v7.5 and Read Free Replication
PPTX
Get More Out of MySQL with TokuDB
PDF
MySQL 5.6 - Operations and Diagnostics Improvements
PDF
Loadays MySQL
PDF
MongoDB .local Toronto 2019: Finding the Right Atlas Cluster Size: Does this ...
PDF
Gruter TECHDAY 2014 Realtime Processing in Telco
PPTX
JSSUG: SQL Sever Performance Tuning
PDF
confessions of a dba: worst and best things I've done in production - Open So...
PDF
All About Storeconfigs
PPTX
Nexcess Magento Imagine 2014 Performance Breakout
PDF
Pluk2013 bodybuilding ratheesh
PPTX
Sql server scalability fundamentals
PDF
Benchmarking, Load Testing, and Preventing Terrible Disasters
PDF
Building a Large Scale SEO/SEM Application with Apache Solr: Presented by Rah...
PDF
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
PPTX
Building a Large Scale SEO/SEM Application with Apache Solr
PPTX
VLDB Administration Strategies
20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02
The InnoDB Storage Engine for MySQL
Rubyslava + PyVo #48
Introduction to TokuDB v7.5 and Read Free Replication
Get More Out of MySQL with TokuDB
MySQL 5.6 - Operations and Diagnostics Improvements
Loadays MySQL
MongoDB .local Toronto 2019: Finding the Right Atlas Cluster Size: Does this ...
Gruter TECHDAY 2014 Realtime Processing in Telco
JSSUG: SQL Sever Performance Tuning
confessions of a dba: worst and best things I've done in production - Open So...
All About Storeconfigs
Nexcess Magento Imagine 2014 Performance Breakout
Pluk2013 bodybuilding ratheesh
Sql server scalability fundamentals
Benchmarking, Load Testing, and Preventing Terrible Disasters
Building a Large Scale SEO/SEM Application with Apache Solr: Presented by Rah...
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
Building a Large Scale SEO/SEM Application with Apache Solr
VLDB Administration Strategies

Recently uploaded (20)

PDF
KodekX | Application Modernization Development
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PDF
Machine learning based COVID-19 study performance prediction
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PDF
Spectral efficient network and resource selection model in 5G networks
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
PPTX
sap open course for s4hana steps from ECC to s4
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PDF
cuic standard and advanced reporting.pdf
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
PDF
Empathic Computing: Creating Shared Understanding
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
KodekX | Application Modernization Development
20250228 LYD VKU AI Blended-Learning.pptx
Machine learning based COVID-19 study performance prediction
Understanding_Digital_Forensics_Presentation.pptx
Per capita expenditure prediction using model stacking based on satellite ima...
Diabetes mellitus diagnosis method based random forest with bat algorithm
Spectral efficient network and resource selection model in 5G networks
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
sap open course for s4hana steps from ECC to s4
“AI and Expert System Decision Support & Business Intelligence Systems”
Advanced methodologies resolving dimensionality complications for autism neur...
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
cuic standard and advanced reporting.pdf
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
Empathic Computing: Creating Shared Understanding
Reach Out and Touch Someone: Haptics and Empathic Computing

Howmysqlworks

  • 2. About me • Before REA I worked for a hosting company for 4 years • Learnt a lot about MySQL during this period • I started as an OPS engineer at REA and did some database stuff here • I havent done much MySQL for two years ! • In this period we’ve had • MySQL 5.6 come out • RDS MySQL improvements ! !
  • 3. Today • High level overview of MySQL • Look inside MySQL server • Talk through how InnoDB performs queries • Schema changes and online schema changes ! !
  • 4. Background • Second biggest relational database in the world • Anyone know what the biggest is? ! • Used by many of the big web shops • Facebook for all feed data • Etsy for user & data • Twitter for tweet persistence • … ! • Sun bought MySQL AB in 2008, people freaked out • Oracle bought Sun in 2010, people freaked out • Big improvements over the past few years (5.5 / 5.6) !
  • 5. How it works mysqld mysql client tcp:3306 filesocket ServerClient libmysql mysql lib ORM host$ mysql host$ mysqld_safe! host$ mysqld
  • 6. Big features • Pluggable storage engines is massive flexibility • You can use a different storage engine per table ! ! • Common storage engines • InnoDB - ACID compliant store • MyISAM - one of the original storage engines - avoid • Lots and lots of others !
  • 7. Inside mysqld mysqld storage engine(s) MyISAM / InnoDB / Memory / etc tcp:3306 filesocket Getting the query and responding bits and bytes stuff API
  • 8. Inside mysqld mysqld storage engine(s) MyISAM / InnoDB / Memory / etc tcp:3306 filesocket API
  • 9. Inside mysqld mysqld storage engine(s) MyISAM / InnoDB / Memory / etc tcp:3306 filesocket Parse query and convert to SE API calls. API
  • 10. Inside mysqld mysqld Storage engines(s) tcp:3306 filesocket API query cache binary log join / sort buffers Binary Log for replication Buffers: Kinda how much batching to storage API to save cost
  • 11. The bad parts - binary log mysqld tcp:3306 filesocket query cache binary log join / sort buffers Because you can mix and match storage engines, replicating state has to be done at the lowest common denominator Innodb Table 1 MyISAM Table 2
  • 12. The bad parts - binary log mysqld Innodb Table 1 tcp:3306 filesocket query cache binary log join / sort buffers Highly concurrent InnoDB needs its queries, serialized down to replicate to slave MyISAM Table 2
  • 13. Compare to other DBs DB server tcp:x filesocket? performance knobs transaction log
  • 14. The bad parts - query cache mysqld Innodb Table 1 tcp:3306 filesocket query cache binary log join / sort buffers Can only work on an entire table level & uses LRU MyISAM Table 2
  • 15. The bad parts - query cache mysqld Innodb Table 1 tcp:3306 filesocket query cache binary log join / sort buffers All ‘update/insert/delete’ blocks all transactions to ensure correct state. It actually slows ! things down MyISAM Table 2
  • 16. Inside mysqld mysqld storage engine(s) MyISAM / InnoDB / Memory / etc tcp:3306 filesocket API Where 10%* of ! the work is done Where 90% ! of the work ! is done *Finger in the air calculation
  • 17. Summary • Turn off query cache! It is hurting you! • Binary log is a necessary evil for replication • Dont play with the performance knobs (sort buffer / join buffer / etc) • Turning the 10% - Focus on the 90% • Use InnoDB for all the tables (unless they’re system tables) ! ! • So…lets look at the 90% ! !
  • 20. Unknown gems mysqld InnoDB tcp:3306 filesocket API handlersocket/ memcache Innodb Memcache API that talks directly to InnoDB ! Removes the SQL layer and is just K/V ! Google: handlersocket memcache InnoDB
  • 21. Key parts of InnoDB Innodb API
  • 22. Key parts of InnoDB Table space (whats on disk) redo log Innodb bufferpool memory! ib_log1! ib_log2! sometable.ibd! ib_data!
  • 23. Key parts of InnoDB Table space redo log bufferpool • Each block is a page • Just think of it as a row with some extra stuff like version number • InnoDB is ACID meaning each connection has to have its own view of the world! • Extra metadata at the top about where everything is
  • 24. Key parts of InnoDB Table space redo log bufferpool • Black is the base metadata
  • 25. What is bufferpool Table space redo log bufferpool • In memory version of hot tablespace pages
  • 26. What is redo log Table space redo log bufferpool • An append only log file of changes
  • 27. What is table space Table space redo log bufferpool • The raw files - but it may not be always up-to-date
  • 28. Lets run some queries Table space redo log bufferpool • First - lets fill up our database with data
  • 29. Lets run some queries Table space redo log bufferpool • mysql> select * from red;
  • 30. Lets run some queries Table space redo log bufferpool • mysql> select * from red; mysqld parses the query and sends ! API calls to InnoDB
  • 31. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Is red in bufferpool?
  • 32. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Load red into buffer pool perform random ! read disk activity! At this point, the time to respond! is dependent on disk speed!
  • 33. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Respond to API calls to mysqld
  • 34. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Parse results ! second pass sort if necessary ! return to client
  • 35. Lets run some queries Table space redo log bufferpool • mysql> select * from red; #again
  • 36. Lets run some queries Table space redo log bufferpool • mysql> select * from red; mysqld parses the query and sends ! API calls to InnoDB
  • 37. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Is red in bufferpool? - yes
  • 38. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Respond to API calls to mysqld
  • 39. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Parse results ! second pass sort if necessary ! return to client
  • 40. Lets run some queries Table space redo log bufferpool • mysql> select * from brown; What if bufferpool is full?
  • 41. Lets run some queries Table space redo log bufferpool • mysql> select * from blue; Assume metadata knows page requirements! LRU on the bufferpool to find space and evict
  • 42. Lets run some queries Table space redo log bufferpool • mysql> select * from blue; ! Load in blue as before and return
  • 43. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Is page in bufferpool? yes
  • 44. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Update the page in bufferpool and increment ! page version number
  • 45. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Write the new page to the redo log! what tablespace is up to!
  • 46. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Update metadata telling it that this is the ! new state of the page! perform sequential ! write disk activity to redo log! This is pretty fast! what tablespace is up to!
  • 47. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Send the OK back to mysqld! what tablespace is up to!
  • 48. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! At sometime in the near future, do a semi sequential ! parse changes and update the tablespace what tablespace is up to! semi sequential write! Not that bad ! !
  • 49. Lets run some queries Table space redo log bufferpool • mysql> insert into yellow values (1,2,3,4); ! Is space in bufferpool?!
  • 50. Lets run some queries Table space redo log bufferpool • mysql> insert into yellow values (1,2,3,4); ! Is adjacent pages in bufferpool?! Random read ! (but not too big)
  • 51. Lets run some queries Table space redo log bufferpool • mysql> insert into yellow values (1,2,3,4); ! Send OK back to client!
  • 52. Key take aways Table space redo log bufferpool • Cold databases are slow! • bufferpool is empty and everything requires random disk reads • The bigger the buffer pool you have, the quicker you will be! (NUMBER 1 factor on performance!)! • Redo log is the second slowest part • we can tune this innodb_flush_logs_at_trx_commit=1 innodb_buffer_pool_size= ~80% memory!
  • 53. Schema changes • Everyone hates them • They are not transactional • They are blocking (well most) • How do they work? !
  • 55. Lets add a few more Storage engines(s) tcp:3306 filesocket API table meta data binary log Table triggers functions / procedures
  • 57. mytbl mysql> CREATE TABLE mytbl 
 (a INT, b CHAR (20), INDEX (a)) ENGINE=InnoDB; Schema changes
  • 58. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT;
  • 59. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Wait for any open transactions to close and then ! lock access to the table! block
  • 60. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Create a new ‘hidden’ table with the new schema! block .mytbl
  • 61. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Do an internal ‘mysqldump’ / ‘mysqlrestore’! block .mytbl
  • 62. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Do a switch-a-roo! block .mytbl
  • 63. Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Unblock and send OK back to client! mytbl
  • 64. Online schema changes • Some changes can be done without ‘locking’ • Column additions • Index additions ! • Any removal cannot be done online • Unless you do it yourself !
  • 66. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… pt-online-schema-change soundclouds lhm etc
  • 67. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl ! Creates another table with the updated schema in the client!
  • 68. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl ! Creates a set of triggers on the main table! triggers
  • 69. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl ! selects the entire table (in batches) and inserts into the new table! ! ! This can take a long time! triggers
  • 70. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers client mysql> select ! Whilst the new table is being built, selects still go to the old table!
  • 71. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers client mysql> UPDATE ! Updates cause the trigger to fire which updates the new table being built. (Inserts the row if its still hasn't been copied)! ! Deletes happen the same way!
  • 72. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers client mysql> INSERT ! Inserts also happen the same way
  • 73. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers ! Eventually the table population finishes and a ‘RENAME table’ action is performed (which is quick)
  • 74. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers ! Eventually the table population finishes. This is the only ‘blocking’ time block block
  • 75. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… ! The schema change tool finishes
  • 76. Online schema changes in 5.6 • All that ‘switch-a-roo’ logic is now within mysqld itself !
  • 77. Summary about schema changes • You ‘double your IO’ when you’re online schema changes! • Binary log is outside of this - so watch out for your slaves table locking if you don't use the right flags ! !