SlideShare a Scribd company logo

Severalnines Self-Training: MySQL® Cluster - Part II

Severalnines, profile picture
Severalnines

Severalnines.com offers free training slides on MySQL Cluster, aiming to share knowledge and skills associated with this technology. The training program includes various modules covering topics such as architecture, installation, performance tuning, and management, along with instructor-led classes for more specialized subjects. Participants should have a basic understanding of SQL and certain system requirements for hands-on exercises.

Technology
Related topics:
Database Management
1 of 54
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
MySQL Cluster Training presented by severalnines.com Address: Contact: SeveralninesAB Jean-Jérôme Schmidt c/o SICS, Box 1263 Email: services@severalnines.com Isafjordsgatan22 SE-164-29 Kista Copyright 2011 Severalnines AB Control your database infrastructure 1
Introduction • At Severalnines, we believe in sharing information and knowledge; we all come from an open source background • We know a lot of things about MySQL Cluster and think that MySQL Cluster is a great technology • These free MySQL Cluster Training slides are a contribution of ours to the knowledge and information sharing that‟s common practice in the open source community • If you have any questions on these slides or would like to book an actual training class, please contact us at: services@severalnines.com Copyright 2011 Severalnines AB Control your database infrastructure 2
Training Slides - Concept • Over the coming weeks we will be chronologically releasing slides for the different sections of our MySQL Cluster Training program on our website. • The full agenda of the training with all of its modules is outlined in the next slides so that you can see what topics will be covered over the coming weeks. • Particularly specialised topics such as Cluster/J or NDB API are not fully covered in the slides. We recommend our instructor-led training classes for such topics. • Please contact us for more details: services@severalnines.com Copyright 2011 Severalnines AB Control your database infrastructure 3
Full Training Agenda (1/4) • MySQL Cluster Introduction – MySQL eco system – Scale up, scale out, and sharding – MySQL Cluster Architecture – Use cases – Features – Node types and Roles • Detailed Concepts – Data Distribution – Verifying data distribution – Access Methods – Partitioning – Node failures and failure detection – Network Partitioning – Transactions and Locking – Consistency Model – Redo logging and Checkpointing Copyright 2011 Severalnines AB Control your database infrastructure 4
Agenda (2/4) • Installing MySQL Cluster – Setting up MySQL Cluster – Starting/stopping nodes – Recovery and restarts – Upgrading configuration – Upgrading Cluster • Performance Tuning (instructor-led only; contact us at services@severalnines.com) – Differences compared to Innodb/MyISAM – Designing efficient and fast applications – Identifying bottlenecks – Tweaking configuration (OS and MySQL Cluster) – Query Tuning – Schema Design – Index Tuning Copyright 2011 Severalnines AB Control your database infrastructure 5
Agenda (3/4) • Management and Administration – Backup and Restore – Geographical Replication – Online and offline operations – Ndbinfo tables – Reporting – Single user mode – Scaling Cluster • Disk Data – Use cases – Limitations – Best practice configuration • Designing a Cluster – Capacity Planning and Dimensioning – Hardware recommendations – Best practice Configuration – Storage calculations Copyright 2011 Severalnines AB Control your database infrastructure 6
Agenda (4/4) • Resolving Issues – Common problems – Error logs and Tracefiles – Recovery and Escalation procedures • Connectivity Overview – NDBAPI – Cluster/J – LDAP • Severalnines Tools – Monitoring and Management – Benchmarking – Sandboxes – Configuration and capacity planning • Conclusion Copyright 2011 Severalnines AB Control your database infrastructure 7
Agenda: Lab Exercises (only applicable to instructor-led training classes) • Lab Exercises – Installing and Loading data into MySQL Cluster – Starting/stopping nodes, recovery – Query tuning – Backup and Restore – Configuration Upgrade • Would you like to try something particular? – This is possible too, speak with your instructor Copyright 2011 Severalnines AB Control your database infrastructure 8
Prerequisites • Readers / Participants have understanding of SQL and basic database concepts. • Laptops/PCs for hands-on exercises • Linux: 1GB RAM • Windows: 2GB RAM • Approx. 20GB disk space and Virtualbox installed. • Virtualbox can be downloaded for free at http://www.virtualbox.org/wiki/Downloads • MySQL Cluster version 7.1 or later Copyright 2011 Severalnines AB Control your database infrastructure 9
Part 2 Section 2: Detailed Concepts Copyright 2011 Severalnines AB Control your database infrastructure 10
Topics covered in Part 2 • Data Distribution & Partitioning • Two Phase Commit Protocol • Transaction Resources Copyright 2011 Severalnines AB Control your database infrastructure 11
Data Distribution and Partitioning Copyright 2011 Severalnines AB Control your database infrastructure 12
MySQL Cluster Architecture Application Application Application R/I/U/D R/I/U/D R/I/U/D Java, C++, LDAP Application R/I/U/D Native API DATA DATA NODE NODE Management Management Server DATA DATA Server NODE NODE Copyright 2011 Severalnines AB Control your database infrastructure 13
Data Distribution • MySQL Cluster automatically partitions tables and distributes the data on the partitions. – It is possible to use user-defined partitioning. • A table is divided into as many partitions as you have data nodes and data is “striped” on the partitions. • Hashing of the Primary Key determines the Partition that should hold the data - automatic distribution. • It is possible to hash on part of the primary key – great for sharding and locality of data! Copyright 2011 Severalnines AB Control your database infrastructure 14
Partitioning and Data Distribution • With two data nodes, data node 1 and data node 2, each table is divided into two Partitions. • Subidis the Primary Key. • Hashing on the Primary Key (subid) determines the Partition. It is also possible to hash on part of the PK. – Odd PKs Green – Even PKs Red STORAGE LAYER subid data DATA DATA NODE 1 NODE 2 1 A Partition 0 3 B 2 C Partition 1 P0 P1 4 D Copyright 2011 Severalnines AB Control your database infrastructure 15
Replicas • To provide for redundancy and fast failover partitions are synchronouslyreplicated. • Two replicas (two copies of data) is most common – it is possible to have one (previous slide), two, three and four replicas. – NoOfReplicas=2 STORAGE LAYER DATA DATA NODE 1 NODE 2 P0 S0 Copyright 2011 Severalnines AB Control your database infrastructure 16
Replicas • To provide for redundancy and fast failover partitions are synchronouslyreplicated from a PRIMARY to a SECONDARY partition. – When a change ( ) is made on P0 it is synchronously replicated to S0. – The change is persisted at transaction COMMIT. – Either both P0 or S0 is update or NOTHING AT ALL STORAGE LAYER DATA DATA NODE 1 NODE 2 P0 S0 Copyright 2011 Severalnines AB Control your database infrastructure 17
Data Distribution - disk logging • After the COMMIT, the data is in main memory. – But changes are REDO LOGGED, but the REDO LOG is synced to disk every xth ms (1000 ms is recommended). • Controlled by TimeBetweenGlobalCheckpoints • Similar to innodb-flush-log-at-trx_commit=2 – The data is also checkpointed to disk. • More on this later. • The disk logging makes it possible to recovery a completely failed cluster. Copyright 2011 Severalnines AB Control your database infrastructure 18
Replicas • To provide for redundancy and fast failover partitions are synchronouslyreplicated. • Two replicas (two copies of data) is most common – it is possible to have one (previous slide), two, three and four replicas. – NoOfReplicas=2 STORAGE LAYER subid data DATA DATA NODE 1 NODE 2 1 A Partition 0 3 B 2 C Partition 1 P0 P1 4 D S1 S0 Px == PRIMARY Partition x Sx == SECONDARY Partition x Copyright 2011 Severalnines AB Control your database infrastructure 19
Node groups • Data Nodes sharing the same data data belongs to a node group. • A node group contains as many nodes as you have replicas (NoOfReplicas=2 is used in the picture below). STORAGE LAYER subid data DATA DATA NODE 1 NODE 2 1 A Partition 0 3 B 2 C Partition 1 P0 P1 4 D S1 S0 Node group 0 Px == PRIMARY Partition x Sx == SECONDARY Parttionx Copyright 2011 Severalnines AB Control your database infrastructure 20
Two Replicas – Four Data Nodes • Four data nodes – four partitions – two replicas • Four data nodes and two replicas  two node groups – No of node groups = #data nodes / #replicas STORAGE LAYER subid data 1 A Partition 0 2 B DATA DATA DATA DATA NODE 1 NODE 2 NODE 3 NODE 4 3 C Partition 1 4 D P0 P1 P2 P3 5 E Partition 2 6 F S1 S0 S3 S2 7 G Partition 3 Node group 0 Node group 1 8 H Px == PRIMARY Partition x Sx == SECONDARY Parttionx Copyright 2011 Severalnines AB Control your database infrastructure 21
Three Replicas – Three Data Nodes • Three data nodes – Three partitions – More replicas gives slower writes. • Three data nodes and three replicas  one node group. This is not common. – Two replicas is common practice/best practice STORAGE LAYER DATA DATA DATA subid data NODE 1 NODE 2 NODE 3 1 A Partition 0 2 B P0 P1 P2 3 C Partition 1 S1 S0 S0 4 D S2 S2 S1 5 E Partition 6 F 2 Node group 0 Px == PRIMARY Partition x Sx == SECONDARY Parttionx Copyright 2011 Severalnines AB Control your database infrastructure 22
Recommendations • Two Replicas recommended - gives best tradeoff between Performance and Availability • Three Replicas (or Four) will give slower writes (, is less tested (not so many deployments), and less deployments are made with this. • Cost of write (approx) – 1 replica (no redundancy): cost X – 2 replicas: cost 2X – 3 replicas: cost 3X Copyright 2011 Severalnines AB Control your database infrastructure 23
Verifying Data Distribution • A helper program called ndb_desc shows how many records are in each partition: ndb_desc –c<mgmt_host:1186> –d<database> -p<tablename> Copyright 2011 Severalnines AB Control your database infrastructure 24
Two Phase Commit Protocol Copyright 2011 Severalnines AB Control your database infrastructure 25
View of the Data Node • Each node has as active Transaction Coordinator (TC). – Every Transaction starts at the TC – One transaction consists of one or more Operations. • Operation is SELECT, INSERT, UPDATE, DELETE • Each node has 1-4 Local Query Handlers (LQH) – Executes the operations TC • ACCess stores a hash table LQH – Primary Key and Unique • TUPle stores data records ACC TUP – ACC hash index links to record in TUP Copyright 2011 Severalnines AB Control your database infrastructure 26
Two Phase Commit Protocol • 2PC facilitates the synchronous replication – To safeguard that both Primary and Secondary Partitions are modified in a Consistent Way – Locking (Row level) • 2PC consists of two phases – Prepare Phase • Grab locks • Make a update to a shadow copy – Commit Phase • Make the shadow copy the real copy • Release locks • 2PC protocol is only involved in – Update, Delete, Insert • Reads are not 2PC (would be very slow and pointless) Copyright 2011 Severalnines AB Control your database infrastructure 27
Two Phase Commit Protocol • INSERT INTO t1(id, data) VALUES (1, „hello‟) – Two data nodes - PK hashes to Primary P0. – The same applies to UPDATES and DELETEs TC TC LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 28
Phase 1 - Prepare Phase • INSERT INTO t1(id, data) VALUES (1, „hello‟) • The protocol starts at the TC that has the Primary Partition for the PK TC TC LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 29
Phase 1 - Prepare Phase • TC starts by calculating a hash (md5sum) on the Entire Primary key. – md5sum(PK) – This gives 128 bit hash value, divided up in 2 x 64-bit INDEX_HASH PARTITION_HASH – Actually further calculations are made • PARTITION= PARTITION_HASH % NO_OF_PARTITIONS • INDEX_POS=INDEX_HASH % NO_OF_BUCKETS – LH*3 Algorithm is what is really being used. • Now MySQL Cluster will know what nodes and partitions are involved. • INSERT INTO t1(id, data) VALUES (1, „hello‟) – INDEX_POS=23 – PARTITION=P0 Copyright 2011 Severalnines AB Control your database infrastructure 30
Phase 1 - Prepare Phase • INSERT INTO t1(id, data) VALUES (1, „hello‟) • Prepare on the local LQH TC TC LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 31
Phase 1 - Prepare Phase • Insert into the hash table in ACC and write record in TUP (allocate space for the record). – The Index Entry is locked now TC TC PREPARE LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello POS=23 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 32
Phase 1 - Prepare Phase • Send PREPARE message to the other LQH for the secondary Partition (S0), and do exactly the same thing. TC TC PREPARE PREPARE LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 33
Phase 1 - Prepare Phase • Send back PREPARE OK to TC TC PREPARE OK TC PREPARE PREPARE LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 34
Phase 2 - Commit Phase • Send COMMIT message to REMOTE LQH first – Release locks, simplifies error handling, ops waiting on lock can resume faster. TC COMMIT TC LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 35
Phase 2 - Commit Phase • Send COMMIT to LQH of P0 – Release locks TC COMMIT TC COMMIT LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 36
Phase 2 - Commit Phase • Send COMMIT OK to TC TC COMMIT TC COMMIT OK COMMIT LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 37
Phase 2 - Commit Phase • Send COMMIT ACK to application COMMIT ACK TC COMMIT TC COMMIT OK COMMIT LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 38
Failure Handling • If the Transaction Coordinator fails another TC will take over. • The new TC will ask the nodes “Send me a list of outstanding transactions from the failed TC” • The new TC will – ABORT transactions in the Prepare Phase – Force COMMIT of transactions IFF another node has seen a COMMIT message. • The Node that has failed will get the changes when it recovers. Copyright 2011 Severalnines AB Control your database infrastructure 39
Transaction Timers • Deadlock detection – waiting for lock – If a transaction is waiting too long time for a lock it is aborted • TransactionDeadLockDetectionTimeout=3000 [ms] • Inactive Transactions – holding a lock – If a transaction is holding a lock too long it is aborted • TransactionInactiveTimeout=60000 [ms] – Never set TransactioInactiveTimeout=0 Copyright 2011 Severalnines AB Control your database infrastructure 40
Locking • Committed Read Isolation level is used • This means that a transaction – Will read last committed data unless it has itself modified this data. • Locks are acquired in the Prepare Phase and released in the Commit Phase. Copyright 2011 Severalnines AB Control your database infrastructure 41
Transaction Resources Copyright 2011 Severalnines AB Control your database infrastructure 42
Transaction and Operation Records • Each transaction requires on Transaction Record (TR) – When it was started – Controlled by MaxNoOfConcurrentTransactions • Also each Operation requires one Operation Record (OR) in TC and one Local Operation Record (LOR) in LQH – What transaction it belongs to – Operation Type – Controlled by MaxNoOfConcurrentOperations – and MaxNoOfLocalOperations Copyright 2011 Severalnines AB Control your database infrastructure 43
Transaction and Operation Records • Assume – MaxNoOfConcurrentTransactions=3 – MaxNoOfConcurrentOperations=3 – MaxNoOfLocalOperations=4 • By default there are 10% more Local Ops than Conc Ops. • Query – INSERT INTO t1 (id,data) VALUES(1,‟x‟); • One transaction • One operation Copyright 2011 Severalnines AB Control your database infrastructure 44
Transaction Starts (prepare phase) • INSERT INTO t1 (id,data) VALUES(1,‟x‟); TR TC TC OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 45
Transaction Starts (prepare phase) • Allocate a Transaction Record at TC TR TC TC OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 46
Transaction Starts (prepare phase) • Allocate a Operation Record at TC TR TC TC OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 47
Transaction Starts (prepare phase) • Allocate a Local Operation Record at LQH TR TC TC OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 48
Transaction Starts (prepare phase) • With many parallel transactions TR TR TC TC OR OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 49
Transaction Starts (prepare phase) • Wrongly dimensioned for failover • Not enough TRs / ORs TR TR TC TC OR OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 50
Transaction and Operation Records • Define – MaxNoOfConcurrentTransactions= 2 x<max_threads> – MaxNoOfConcurrentOperations= MaxConcurrentTransactionsx<ops in trans> – MaxNoOfLocalOperations=1.1 x MaxNoOfConcurrentOperations • 1.1x is by default • Seldom needed to change this. Copyright 2011 Severalnines AB Control your database infrastructure 51
Coming next in Part 3: Detailed Concepts Access Methods Redo Logging and Check-pointing Recovery Node Failures and Error Detection Network Partitioning Copyright 2011 Severalnines AB Control your database infrastructure 52
We hope these training slides are useful to you! Please visit our website to view the next section of this training. For any questions, comments, feedback or to book a training class, please contact us at: services@severalnines.com Thank you! Copyright 2011 Severalnines AB Control your database infrastructure 53
Disclaimer © Copyright 2011 Severalnines AB. All rights reserved. Severalnines& the Severalnineslogo(s) are trademarks of Severalnines AB. MySQL is a registered trademark of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Copyright 2011 Severalnines AB Control your database infrastructure 54

More Related Content

PPTX
Severalnines Self-Training: MySQL® Cluster - Part V
Severalnines
 
PPTX
Severalnines Self-Training: MySQL® Cluster - Part VII
Severalnines
 
PPTX
Severalnines Self-Training: MySQL® Cluster - Part VIII
Severalnines
 
PPTX
Severalnines Training: MySQL® Cluster - Part IX
Severalnines
 
PPTX
Severalnines Training: MySQL Cluster - Part X
Severalnines
 
PPTX
Severalnines Self-Training: MySQL® Cluster - Part VI
Severalnines
 
PDF
Conference tutorial: MySQL Cluster as NoSQL
Severalnines
 
PDF
My sql5.7 whatsnew_presentedatgids2015
Sanjay Manwani
 
Severalnines Self-Training: MySQL® Cluster - Part V
Severalnines
 
Severalnines Self-Training: MySQL® Cluster - Part VII
Severalnines
 
Severalnines Self-Training: MySQL® Cluster - Part VIII
Severalnines
 
Severalnines Training: MySQL® Cluster - Part IX
Severalnines
 
Severalnines Training: MySQL Cluster - Part X
Severalnines
 
Severalnines Self-Training: MySQL® Cluster - Part VI
Severalnines
 
Conference tutorial: MySQL Cluster as NoSQL
Severalnines
 
My sql5.7 whatsnew_presentedatgids2015
Sanjay Manwani
 

What's hot (20)

PPTX
Ibm spectrum scale fundamentals workshop for americas part 5 ess gnr-usecases...
xKinAnx
 
PDF
MySQL :What's New #GIDS16
Sanjay Manwani
 
PDF
MySQL Performance Tuning
FromDual GmbH
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 2 IBM Spectrum Sca...
xKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 3 Information Life...
xKinAnx
 
PDF
Database & Technology 2 _ Marting Lambert _ Mixed Workloads Why and How.pdf
InSync2011
 
PPT
MySQL Cluster Basics
Wagner Bianchi
 
PPTX
Ibm spectrum scale_backup_n_archive_v03_ash
Ashutosh Mate
 
PDF
MySQL Enterprise Backup
Mario Beck
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 1 components archi...
xKinAnx
 
PDF
MySQL Performance Tuning: The Perfect Scalability (OOW2019)
Mirko Ortensi
 
PPTX
Spectrum Scale - Diversified analytic solution based on various storage servi...
Wei Gong
 
PPTX
Exadata Backup
Fran Navarro
 
PDF
Database as a Service on the Oracle Database Appliance Platform
Maris Elsins
 
PDF
Intro to GlusterFS Webinar - August 2011
GlusterFS
 
PDF
Webinar Sept 22: Gluster Partners with Redapt to Deliver Scale-Out NAS Storage
GlusterFS
 
PDF
Best Practices of HA and Replication of PostgreSQL in Virtualized Environments
Jignesh Shah
 
PDF
Best Practices for Becoming an Exceptional Postgres DBA
EDB
 
PDF
PostgreSQL and Benchmarks
Jignesh Shah
 
PDF
IBM Spectrum Scale for File and Object Storage
Tony Pearson
 
Ibm spectrum scale fundamentals workshop for americas part 5 ess gnr-usecases...
xKinAnx
 
MySQL :What's New #GIDS16
Sanjay Manwani
 
MySQL Performance Tuning
FromDual GmbH
 
Ibm spectrum scale fundamentals workshop for americas part 2 IBM Spectrum Sca...
xKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 3 Information Life...
xKinAnx
 
Database & Technology 2 _ Marting Lambert _ Mixed Workloads Why and How.pdf
InSync2011
 
MySQL Cluster Basics
Wagner Bianchi
 
Ibm spectrum scale_backup_n_archive_v03_ash
Ashutosh Mate
 
MySQL Enterprise Backup
Mario Beck
 
Ibm spectrum scale fundamentals workshop for americas part 1 components archi...
xKinAnx
 
MySQL Performance Tuning: The Perfect Scalability (OOW2019)
Mirko Ortensi
 
Spectrum Scale - Diversified analytic solution based on various storage servi...
Wei Gong
 
Exadata Backup
Fran Navarro
 
Database as a Service on the Oracle Database Appliance Platform
Maris Elsins
 
Intro to GlusterFS Webinar - August 2011
GlusterFS
 
Webinar Sept 22: Gluster Partners with Redapt to Deliver Scale-Out NAS Storage
GlusterFS
 
Best Practices of HA and Replication of PostgreSQL in Virtualized Environments
Jignesh Shah
 
Best Practices for Becoming an Exceptional Postgres DBA
EDB
 
PostgreSQL and Benchmarks
Jignesh Shah
 
IBM Spectrum Scale for File and Object Storage
Tony Pearson
 
Ad

Similar to Severalnines Self-Training: MySQL® Cluster - Part II (20)

PDF
Santo Leto - MySQL Connect 2012 - Getting Started with Mysql Cluster
Santo Leto
 
KEY
Cloud Computing & Scaling Web Apps
Mark Slingsby
 
PDF
Oracle my sql cluster cge
seungdon1
 
PPTX
Handling Massive Writes
Liran Zelkha
 
PPT
MySQL 5.6 Updates
Dave Stokes
 
PDF
Oracle MySQL Tutorial -- MySQL NoSQL Cloud Buenos Aires Nov, 13 2014
Manuel Contreras
 
PDF
02 -my_sql_roma-may2011
testfank
 
PDF
MySQL for Oracle DBAs
Mario Beck
 
PDF
Fb talk arch_summit
drewz lin
 
PDF
Lessons Learned: Novell Open Enterprise Server Upgrades Made Easy
Novell
 
PDF
MySQL Intro JSON NoSQL
Mark Swarbrick
 
PDF
Novell File Management Suite Use Cases
Novell
 
PDF
Novell File Management Suite Use Cases
Novell
 
PPTX
Cassandra EU 2012 - Overview of Case Studies and State of the Market by 451 R...
Acunu
 
PDF
Storage Systems For Scalable systems
elliando dias
 
PDF
MySQL高可用
thinkinlamp
 
PDF
Novell Storage Manager: Your Secret Weapon for Simplified File and User Manag...
Novell
 
PDF
Oracle to MySQL 2012
Marco Tusa
 
PDF
Avi Jain - Resume Descriptive 2015
AVI JAIN
 
PDF
Avi Jain - Resume Descriptive 2015
AVI JAIN
 
Santo Leto - MySQL Connect 2012 - Getting Started with Mysql Cluster
Santo Leto
 
Cloud Computing & Scaling Web Apps
Mark Slingsby
 
Oracle my sql cluster cge
seungdon1
 
Handling Massive Writes
Liran Zelkha
 
MySQL 5.6 Updates
Dave Stokes
 
Oracle MySQL Tutorial -- MySQL NoSQL Cloud Buenos Aires Nov, 13 2014
Manuel Contreras
 
02 -my_sql_roma-may2011
testfank
 
MySQL for Oracle DBAs
Mario Beck
 
Fb talk arch_summit
drewz lin
 
Lessons Learned: Novell Open Enterprise Server Upgrades Made Easy
Novell
 
MySQL Intro JSON NoSQL
Mark Swarbrick
 
Novell File Management Suite Use Cases
Novell
 
Novell File Management Suite Use Cases
Novell
 
Cassandra EU 2012 - Overview of Case Studies and State of the Market by 451 R...
Acunu
 
Storage Systems For Scalable systems
elliando dias
 
MySQL高可用
thinkinlamp
 
Novell Storage Manager: Your Secret Weapon for Simplified File and User Manag...
Novell
 
Oracle to MySQL 2012
Marco Tusa
 
Avi Jain - Resume Descriptive 2015
AVI JAIN
 
Avi Jain - Resume Descriptive 2015
AVI JAIN
 
Ad

More from Severalnines (20)

PDF
The Long Term Cost of Managed DBaaS vs Sovereign DBaaS
Severalnines
 
PPTX
Sovereign DBaaS_ A Practical Vision for Self-Implementation of DBaaS.pptx
Severalnines
 
PDF
PostgreSQL on AWS Aurora/Azure Cosmos VS EC2/Azure VMs
Severalnines
 
PDF
Localhost Conference 2024_ Building a Flexible and Scalable Database Strategy...
Severalnines
 
PDF
SREDAY London 2024 | Cloud Native Technologies: The Building Blocks of Modern...
Severalnines
 
PDF
Building a Sovereign DBaaS on K8s OpenInfra Summit Asia 2024.pdf
Severalnines
 
PDF
S-DBaaS Community Call | Introduction to Sovereign DBaaS: The why, what and how
Severalnines
 
PDF
WEBINAR SLIDES: CCX for Cloud Service Providers
Severalnines
 
PPTX
LIVE DEMO: CCX for CSPs, a drop-in DBaaS solution
Severalnines
 
PDF
Kubernetes at Scale: Going Multi-Cluster with Istio
Severalnines
 
PDF
DIY DBaaS: A guide to building your own full-featured DBaaS
Severalnines
 
PDF
Cloud's future runs through Sovereign DBaaS
Severalnines
 
PPTX
Tips to drive maria db cluster performance for nextcloud
Severalnines
 
PPTX
Working with the Moodle Database: The Basics
Severalnines
 
PPTX
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
Severalnines
 
PDF
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
Severalnines
 
PDF
Webinar slides: How to Migrate from Oracle DB to MariaDB
Severalnines
 
PDF
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Severalnines
 
PDF
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Severalnines
 
PDF
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Severalnines
 
The Long Term Cost of Managed DBaaS vs Sovereign DBaaS
Severalnines
 
Sovereign DBaaS_ A Practical Vision for Self-Implementation of DBaaS.pptx
Severalnines
 
PostgreSQL on AWS Aurora/Azure Cosmos VS EC2/Azure VMs
Severalnines
 
Localhost Conference 2024_ Building a Flexible and Scalable Database Strategy...
Severalnines
 
SREDAY London 2024 | Cloud Native Technologies: The Building Blocks of Modern...
Severalnines
 
Building a Sovereign DBaaS on K8s OpenInfra Summit Asia 2024.pdf
Severalnines
 
S-DBaaS Community Call | Introduction to Sovereign DBaaS: The why, what and how
Severalnines
 
WEBINAR SLIDES: CCX for Cloud Service Providers
Severalnines
 
LIVE DEMO: CCX for CSPs, a drop-in DBaaS solution
Severalnines
 
Kubernetes at Scale: Going Multi-Cluster with Istio
Severalnines
 
DIY DBaaS: A guide to building your own full-featured DBaaS
Severalnines
 
Cloud's future runs through Sovereign DBaaS
Severalnines
 
Tips to drive maria db cluster performance for nextcloud
Severalnines
 
Working with the Moodle Database: The Basics
Severalnines
 
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
Severalnines
 
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
Severalnines
 
Webinar slides: How to Migrate from Oracle DB to MariaDB
Severalnines
 
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Severalnines
 
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Severalnines
 
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Severalnines
 

Recently uploaded (20)

PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
KodekX | Application Modernization Development
KodekX
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
KodekX | Application Modernization Development
KodekX
 
Cloud computing and distributed systems.
kkkkkhan
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
Encapsulation theory and applications.pdf
gurumoop
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 

Severalnines Self-Training: MySQL® Cluster - Part II

  • 1. MySQL Cluster Training presented by severalnines.com Address: Contact: SeveralninesAB Jean-Jérôme Schmidt c/o SICS, Box 1263 Email: services@severalnines.com Isafjordsgatan22 SE-164-29 Kista Copyright 2011 Severalnines AB Control your database infrastructure 1
  • 2. Introduction • At Severalnines, we believe in sharing information and knowledge; we all come from an open source background • We know a lot of things about MySQL Cluster and think that MySQL Cluster is a great technology • These free MySQL Cluster Training slides are a contribution of ours to the knowledge and information sharing that‟s common practice in the open source community • If you have any questions on these slides or would like to book an actual training class, please contact us at: services@severalnines.com Copyright 2011 Severalnines AB Control your database infrastructure 2
  • 3. Training Slides - Concept • Over the coming weeks we will be chronologically releasing slides for the different sections of our MySQL Cluster Training program on our website. • The full agenda of the training with all of its modules is outlined in the next slides so that you can see what topics will be covered over the coming weeks. • Particularly specialised topics such as Cluster/J or NDB API are not fully covered in the slides. We recommend our instructor-led training classes for such topics. • Please contact us for more details: services@severalnines.com Copyright 2011 Severalnines AB Control your database infrastructure 3
  • 4. Full Training Agenda (1/4) • MySQL Cluster Introduction – MySQL eco system – Scale up, scale out, and sharding – MySQL Cluster Architecture – Use cases – Features – Node types and Roles • Detailed Concepts – Data Distribution – Verifying data distribution – Access Methods – Partitioning – Node failures and failure detection – Network Partitioning – Transactions and Locking – Consistency Model – Redo logging and Checkpointing Copyright 2011 Severalnines AB Control your database infrastructure 4
  • 5. Agenda (2/4) • Installing MySQL Cluster – Setting up MySQL Cluster – Starting/stopping nodes – Recovery and restarts – Upgrading configuration – Upgrading Cluster • Performance Tuning (instructor-led only; contact us at services@severalnines.com) – Differences compared to Innodb/MyISAM – Designing efficient and fast applications – Identifying bottlenecks – Tweaking configuration (OS and MySQL Cluster) – Query Tuning – Schema Design – Index Tuning Copyright 2011 Severalnines AB Control your database infrastructure 5
  • 6. Agenda (3/4) • Management and Administration – Backup and Restore – Geographical Replication – Online and offline operations – Ndbinfo tables – Reporting – Single user mode – Scaling Cluster • Disk Data – Use cases – Limitations – Best practice configuration • Designing a Cluster – Capacity Planning and Dimensioning – Hardware recommendations – Best practice Configuration – Storage calculations Copyright 2011 Severalnines AB Control your database infrastructure 6
  • 7. Agenda (4/4) • Resolving Issues – Common problems – Error logs and Tracefiles – Recovery and Escalation procedures • Connectivity Overview – NDBAPI – Cluster/J – LDAP • Severalnines Tools – Monitoring and Management – Benchmarking – Sandboxes – Configuration and capacity planning • Conclusion Copyright 2011 Severalnines AB Control your database infrastructure 7
  • 8. Agenda: Lab Exercises (only applicable to instructor-led training classes) • Lab Exercises – Installing and Loading data into MySQL Cluster – Starting/stopping nodes, recovery – Query tuning – Backup and Restore – Configuration Upgrade • Would you like to try something particular? – This is possible too, speak with your instructor Copyright 2011 Severalnines AB Control your database infrastructure 8
  • 9. Prerequisites • Readers / Participants have understanding of SQL and basic database concepts. • Laptops/PCs for hands-on exercises • Linux: 1GB RAM • Windows: 2GB RAM • Approx. 20GB disk space and Virtualbox installed. • Virtualbox can be downloaded for free at http://www.virtualbox.org/wiki/Downloads • MySQL Cluster version 7.1 or later Copyright 2011 Severalnines AB Control your database infrastructure 9
  • 10. Part 2 Section 2: Detailed Concepts Copyright 2011 Severalnines AB Control your database infrastructure 10
  • 11. Topics covered in Part 2 • Data Distribution & Partitioning • Two Phase Commit Protocol • Transaction Resources Copyright 2011 Severalnines AB Control your database infrastructure 11
  • 12. Data Distribution and Partitioning Copyright 2011 Severalnines AB Control your database infrastructure 12
  • 13. MySQL Cluster Architecture Application Application Application R/I/U/D R/I/U/D R/I/U/D Java, C++, LDAP Application R/I/U/D Native API DATA DATA NODE NODE Management Management Server DATA DATA Server NODE NODE Copyright 2011 Severalnines AB Control your database infrastructure 13
  • 14. Data Distribution • MySQL Cluster automatically partitions tables and distributes the data on the partitions. – It is possible to use user-defined partitioning. • A table is divided into as many partitions as you have data nodes and data is “striped” on the partitions. • Hashing of the Primary Key determines the Partition that should hold the data - automatic distribution. • It is possible to hash on part of the primary key – great for sharding and locality of data! Copyright 2011 Severalnines AB Control your database infrastructure 14
  • 15. Partitioning and Data Distribution • With two data nodes, data node 1 and data node 2, each table is divided into two Partitions. • Subidis the Primary Key. • Hashing on the Primary Key (subid) determines the Partition. It is also possible to hash on part of the PK. – Odd PKs Green – Even PKs Red STORAGE LAYER subid data DATA DATA NODE 1 NODE 2 1 A Partition 0 3 B 2 C Partition 1 P0 P1 4 D Copyright 2011 Severalnines AB Control your database infrastructure 15
  • 16. Replicas • To provide for redundancy and fast failover partitions are synchronouslyreplicated. • Two replicas (two copies of data) is most common – it is possible to have one (previous slide), two, three and four replicas. – NoOfReplicas=2 STORAGE LAYER DATA DATA NODE 1 NODE 2 P0 S0 Copyright 2011 Severalnines AB Control your database infrastructure 16
  • 17. Replicas • To provide for redundancy and fast failover partitions are synchronouslyreplicated from a PRIMARY to a SECONDARY partition. – When a change ( ) is made on P0 it is synchronously replicated to S0. – The change is persisted at transaction COMMIT. – Either both P0 or S0 is update or NOTHING AT ALL STORAGE LAYER DATA DATA NODE 1 NODE 2 P0 S0 Copyright 2011 Severalnines AB Control your database infrastructure 17
  • 18. Data Distribution - disk logging • After the COMMIT, the data is in main memory. – But changes are REDO LOGGED, but the REDO LOG is synced to disk every xth ms (1000 ms is recommended). • Controlled by TimeBetweenGlobalCheckpoints • Similar to innodb-flush-log-at-trx_commit=2 – The data is also checkpointed to disk. • More on this later. • The disk logging makes it possible to recovery a completely failed cluster. Copyright 2011 Severalnines AB Control your database infrastructure 18
  • 19. Replicas • To provide for redundancy and fast failover partitions are synchronouslyreplicated. • Two replicas (two copies of data) is most common – it is possible to have one (previous slide), two, three and four replicas. – NoOfReplicas=2 STORAGE LAYER subid data DATA DATA NODE 1 NODE 2 1 A Partition 0 3 B 2 C Partition 1 P0 P1 4 D S1 S0 Px == PRIMARY Partition x Sx == SECONDARY Partition x Copyright 2011 Severalnines AB Control your database infrastructure 19
  • 20. Node groups • Data Nodes sharing the same data data belongs to a node group. • A node group contains as many nodes as you have replicas (NoOfReplicas=2 is used in the picture below). STORAGE LAYER subid data DATA DATA NODE 1 NODE 2 1 A Partition 0 3 B 2 C Partition 1 P0 P1 4 D S1 S0 Node group 0 Px == PRIMARY Partition x Sx == SECONDARY Parttionx Copyright 2011 Severalnines AB Control your database infrastructure 20
  • 21. Two Replicas – Four Data Nodes • Four data nodes – four partitions – two replicas • Four data nodes and two replicas  two node groups – No of node groups = #data nodes / #replicas STORAGE LAYER subid data 1 A Partition 0 2 B DATA DATA DATA DATA NODE 1 NODE 2 NODE 3 NODE 4 3 C Partition 1 4 D P0 P1 P2 P3 5 E Partition 2 6 F S1 S0 S3 S2 7 G Partition 3 Node group 0 Node group 1 8 H Px == PRIMARY Partition x Sx == SECONDARY Parttionx Copyright 2011 Severalnines AB Control your database infrastructure 21
  • 22. Three Replicas – Three Data Nodes • Three data nodes – Three partitions – More replicas gives slower writes. • Three data nodes and three replicas  one node group. This is not common. – Two replicas is common practice/best practice STORAGE LAYER DATA DATA DATA subid data NODE 1 NODE 2 NODE 3 1 A Partition 0 2 B P0 P1 P2 3 C Partition 1 S1 S0 S0 4 D S2 S2 S1 5 E Partition 6 F 2 Node group 0 Px == PRIMARY Partition x Sx == SECONDARY Parttionx Copyright 2011 Severalnines AB Control your database infrastructure 22
  • 23. Recommendations • Two Replicas recommended - gives best tradeoff between Performance and Availability • Three Replicas (or Four) will give slower writes (, is less tested (not so many deployments), and less deployments are made with this. • Cost of write (approx) – 1 replica (no redundancy): cost X – 2 replicas: cost 2X – 3 replicas: cost 3X Copyright 2011 Severalnines AB Control your database infrastructure 23
  • 24. Verifying Data Distribution • A helper program called ndb_desc shows how many records are in each partition: ndb_desc –c<mgmt_host:1186> –d<database> -p<tablename> Copyright 2011 Severalnines AB Control your database infrastructure 24
  • 25. Two Phase Commit Protocol Copyright 2011 Severalnines AB Control your database infrastructure 25
  • 26. View of the Data Node • Each node has as active Transaction Coordinator (TC). – Every Transaction starts at the TC – One transaction consists of one or more Operations. • Operation is SELECT, INSERT, UPDATE, DELETE • Each node has 1-4 Local Query Handlers (LQH) – Executes the operations TC • ACCess stores a hash table LQH – Primary Key and Unique • TUPle stores data records ACC TUP – ACC hash index links to record in TUP Copyright 2011 Severalnines AB Control your database infrastructure 26
  • 27. Two Phase Commit Protocol • 2PC facilitates the synchronous replication – To safeguard that both Primary and Secondary Partitions are modified in a Consistent Way – Locking (Row level) • 2PC consists of two phases – Prepare Phase • Grab locks • Make a update to a shadow copy – Commit Phase • Make the shadow copy the real copy • Release locks • 2PC protocol is only involved in – Update, Delete, Insert • Reads are not 2PC (would be very slow and pointless) Copyright 2011 Severalnines AB Control your database infrastructure 27
  • 28. Two Phase Commit Protocol • INSERT INTO t1(id, data) VALUES (1, „hello‟) – Two data nodes - PK hashes to Primary P0. – The same applies to UPDATES and DELETEs TC TC LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 28
  • 29. Phase 1 - Prepare Phase • INSERT INTO t1(id, data) VALUES (1, „hello‟) • The protocol starts at the TC that has the Primary Partition for the PK TC TC LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 29
  • 30. Phase 1 - Prepare Phase • TC starts by calculating a hash (md5sum) on the Entire Primary key. – md5sum(PK) – This gives 128 bit hash value, divided up in 2 x 64-bit INDEX_HASH PARTITION_HASH – Actually further calculations are made • PARTITION= PARTITION_HASH % NO_OF_PARTITIONS • INDEX_POS=INDEX_HASH % NO_OF_BUCKETS – LH*3 Algorithm is what is really being used. • Now MySQL Cluster will know what nodes and partitions are involved. • INSERT INTO t1(id, data) VALUES (1, „hello‟) – INDEX_POS=23 – PARTITION=P0 Copyright 2011 Severalnines AB Control your database infrastructure 30
  • 31. Phase 1 - Prepare Phase • INSERT INTO t1(id, data) VALUES (1, „hello‟) • Prepare on the local LQH TC TC LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 31
  • 32. Phase 1 - Prepare Phase • Insert into the hash table in ACC and write record in TUP (allocate space for the record). – The Index Entry is locked now TC TC PREPARE LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello POS=23 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 32
  • 33. Phase 1 - Prepare Phase • Send PREPARE message to the other LQH for the secondary Partition (S0), and do exactly the same thing. TC TC PREPARE PREPARE LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 33
  • 34. Phase 1 - Prepare Phase • Send back PREPARE OK to TC TC PREPARE OK TC PREPARE PREPARE LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 34
  • 35. Phase 2 - Commit Phase • Send COMMIT message to REMOTE LQH first – Release locks, simplifies error handling, ops waiting on lock can resume faster. TC COMMIT TC LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 35
  • 36. Phase 2 - Commit Phase • Send COMMIT to LQH of P0 – Release locks TC COMMIT TC COMMIT LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 36
  • 37. Phase 2 - Commit Phase • Send COMMIT OK to TC TC COMMIT TC COMMIT OK COMMIT LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 37
  • 38. Phase 2 - Commit Phase • Send COMMIT ACK to application COMMIT ACK TC COMMIT TC COMMIT OK COMMIT LQH LQH ACC TUP ACC TUP P0 P0 S0 S0 1 hello 1 hello IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 38
  • 39. Failure Handling • If the Transaction Coordinator fails another TC will take over. • The new TC will ask the nodes “Send me a list of outstanding transactions from the failed TC” • The new TC will – ABORT transactions in the Prepare Phase – Force COMMIT of transactions IFF another node has seen a COMMIT message. • The Node that has failed will get the changes when it recovers. Copyright 2011 Severalnines AB Control your database infrastructure 39
  • 40. Transaction Timers • Deadlock detection – waiting for lock – If a transaction is waiting too long time for a lock it is aborted • TransactionDeadLockDetectionTimeout=3000 [ms] • Inactive Transactions – holding a lock – If a transaction is holding a lock too long it is aborted • TransactionInactiveTimeout=60000 [ms] – Never set TransactioInactiveTimeout=0 Copyright 2011 Severalnines AB Control your database infrastructure 40
  • 41. Locking • Committed Read Isolation level is used • This means that a transaction – Will read last committed data unless it has itself modified this data. • Locks are acquired in the Prepare Phase and released in the Commit Phase. Copyright 2011 Severalnines AB Control your database infrastructure 41
  • 42. Transaction Resources Copyright 2011 Severalnines AB Control your database infrastructure 42
  • 43. Transaction and Operation Records • Each transaction requires on Transaction Record (TR) – When it was started – Controlled by MaxNoOfConcurrentTransactions • Also each Operation requires one Operation Record (OR) in TC and one Local Operation Record (LOR) in LQH – What transaction it belongs to – Operation Type – Controlled by MaxNoOfConcurrentOperations – and MaxNoOfLocalOperations Copyright 2011 Severalnines AB Control your database infrastructure 43
  • 44. Transaction and Operation Records • Assume – MaxNoOfConcurrentTransactions=3 – MaxNoOfConcurrentOperations=3 – MaxNoOfLocalOperations=4 • By default there are 10% more Local Ops than Conc Ops. • Query – INSERT INTO t1 (id,data) VALUES(1,‟x‟); • One transaction • One operation Copyright 2011 Severalnines AB Control your database infrastructure 44
  • 45. Transaction Starts (prepare phase) • INSERT INTO t1 (id,data) VALUES(1,‟x‟); TR TC TC OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 45
  • 46. Transaction Starts (prepare phase) • Allocate a Transaction Record at TC TR TC TC OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 46
  • 47. Transaction Starts (prepare phase) • Allocate a Operation Record at TC TR TC TC OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 47
  • 48. Transaction Starts (prepare phase) • Allocate a Local Operation Record at LQH TR TC TC OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 48
  • 49. Transaction Starts (prepare phase) • With many parallel transactions TR TR TC TC OR OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 49
  • 50. Transaction Starts (prepare phase) • Wrongly dimensioned for failover • Not enough TRs / ORs TR TR TC TC OR OR PREPARE PREPARE LOR LQH LQH LOR ACC TUP ACC TUP P0 P0 S0 S0 IndexMemory DataMemory IndexMemory DataMemory DATA NODE 1 DATA NODE 2 Copyright 2011 Severalnines AB Control your database infrastructure 50
  • 51. Transaction and Operation Records • Define – MaxNoOfConcurrentTransactions= 2 x<max_threads> – MaxNoOfConcurrentOperations= MaxConcurrentTransactionsx<ops in trans> – MaxNoOfLocalOperations=1.1 x MaxNoOfConcurrentOperations • 1.1x is by default • Seldom needed to change this. Copyright 2011 Severalnines AB Control your database infrastructure 51
  • 52. Coming next in Part 3: Detailed Concepts Access Methods Redo Logging and Check-pointing Recovery Node Failures and Error Detection Network Partitioning Copyright 2011 Severalnines AB Control your database infrastructure 52
  • 53. We hope these training slides are useful to you! Please visit our website to view the next section of this training. For any questions, comments, feedback or to book a training class, please contact us at: services@severalnines.com Thank you! Copyright 2011 Severalnines AB Control your database infrastructure 53
  • 54. Disclaimer © Copyright 2011 Severalnines AB. All rights reserved. Severalnines& the Severalnineslogo(s) are trademarks of Severalnines AB. MySQL is a registered trademark of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Copyright 2011 Severalnines AB Control your database infrastructure 54

Editor's Notes

  • #13: Dear Jury, Just as the electric grid revolutionized access to electricity 100 years ago, we at Severalnines believe that Cloud Computing will revolutionize IT where organizations will be able to plug into extremely powerful computing resources over the network. We have already seen the beginnings of this new wave, where the current infrastructure stack is being challenged and disrupted by a whole set of new technologies. For instance, in the database market, over 40 startups have received funding over the past 18 months. Severalnines is not building yet another database product, we believe there are already a lot of good technologies available. To manage a database costs 4 times the purchase price, and yet, very few companies are addressing this problem. Severalnines focuses on solutions to address this underserved segment. The founders of the company have a solid background in databases, having been at MySQL since 2003. The company develops a management platform which is database and cloud agnostic.We are database independent since we do not know who, if anybody, will be the next MySQL of the cloud.We are cloud independent, since we do not want to depend on any cloud vendor (e.g. Amazon or Rackspace) to avoid vendor lock-in. After the Amazon EC2 downtime during the Easter break, hundreds of affected companies have realized the importance of this. There is also a commercial aspect for avoiding vendor lock-in. Severalnines enhances productivity of organizations by attacking the biggest cost associated with database systems. We are a Swedish startup, hosted by SICS in Kista. There is also a small but very efficient development capacity in Singapore. Just as MySQL became a major brand and placed Sweden on the global software infrastructure map, we believe Severalnines can become a serious global player in the emerging Cloud space. We are very thankful that an organization like Eurocloud exists, and would like to thank the jury for considering our application. Kind regards,Vinay Joosery Severalnines AB
  • #15: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #16: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #17: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #18: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #19: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #20: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #21: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #22: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #23: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #24: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #25: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #26: Dear Jury, Just as the electric grid revolutionized access to electricity 100 years ago, we at Severalnines believe that Cloud Computing will revolutionize IT where organizations will be able to plug into extremely powerful computing resources over the network. We have already seen the beginnings of this new wave, where the current infrastructure stack is being challenged and disrupted by a whole set of new technologies. For instance, in the database market, over 40 startups have received funding over the past 18 months. Severalnines is not building yet another database product, we believe there are already a lot of good technologies available. To manage a database costs 4 times the purchase price, and yet, very few companies are addressing this problem. Severalnines focuses on solutions to address this underserved segment. The founders of the company have a solid background in databases, having been at MySQL since 2003. The company develops a management platform which is database and cloud agnostic.We are database independent since we do not know who, if anybody, will be the next MySQL of the cloud.We are cloud independent, since we do not want to depend on any cloud vendor (e.g. Amazon or Rackspace) to avoid vendor lock-in. After the Amazon EC2 downtime during the Easter break, hundreds of affected companies have realized the importance of this. There is also a commercial aspect for avoiding vendor lock-in. Severalnines enhances productivity of organizations by attacking the biggest cost associated with database systems. We are a Swedish startup, hosted by SICS in Kista. There is also a small but very efficient development capacity in Singapore. Just as MySQL became a major brand and placed Sweden on the global software infrastructure map, we believe Severalnines can become a serious global player in the emerging Cloud space. We are very thankful that an organization like Eurocloud exists, and would like to thank the jury for considering our application. Kind regards,Vinay Joosery Severalnines AB
  • #27: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #28: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #29: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #30: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #31: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #32: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #33: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #34: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #35: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #36: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #37: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #38: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #39: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #40: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #41: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #42: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #43: Dear Jury, Just as the electric grid revolutionized access to electricity 100 years ago, we at Severalnines believe that Cloud Computing will revolutionize IT where organizations will be able to plug into extremely powerful computing resources over the network. We have already seen the beginnings of this new wave, where the current infrastructure stack is being challenged and disrupted by a whole set of new technologies. For instance, in the database market, over 40 startups have received funding over the past 18 months. Severalnines is not building yet another database product, we believe there are already a lot of good technologies available. To manage a database costs 4 times the purchase price, and yet, very few companies are addressing this problem. Severalnines focuses on solutions to address this underserved segment. The founders of the company have a solid background in databases, having been at MySQL since 2003. The company develops a management platform which is database and cloud agnostic.We are database independent since we do not know who, if anybody, will be the next MySQL of the cloud.We are cloud independent, since we do not want to depend on any cloud vendor (e.g. Amazon or Rackspace) to avoid vendor lock-in. After the Amazon EC2 downtime during the Easter break, hundreds of affected companies have realized the importance of this. There is also a commercial aspect for avoiding vendor lock-in. Severalnines enhances productivity of organizations by attacking the biggest cost associated with database systems. We are a Swedish startup, hosted by SICS in Kista. There is also a small but very efficient development capacity in Singapore. Just as MySQL became a major brand and placed Sweden on the global software infrastructure map, we believe Severalnines can become a serious global player in the emerging Cloud space. We are very thankful that an organization like Eurocloud exists, and would like to thank the jury for considering our application. Kind regards,Vinay Joosery Severalnines AB
  • #44: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #45: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #46: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #47: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #48: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #49: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #50: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #51: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com
  • #55: Severalnines has been offering its products free of charge since 2007, while the founders were employed at MySQL. These products are the de-facto standard tools to assist MySQL customers and users in deploying their MySQL clusters. More information about Severalnines at www.severalnines.com