Continuent Tungsten - Scalable Saa S Data Management

Scalable SaaS Data Management with Tungsten Robert Hodges CTO, Continuent, Inc.

Agenda Introductions What’s So Hard about SaaS Data Management? Introducing Tungsten Clustering Implementing SaaS Data Management with Tungsten Commercial Support for SaaS from Continuent Definition: SaaS = “Software as a Service”

About Continuent Our Business: Full service database management for SaaS applications Our Value: Ensure data are available when and where you need them TCO less than 20% of comparable solutions Our Technical Expertise Database replication Cluster management Application connectivity SaaS applications

What’s So Hard about SaaS Data Management?

The SaaS Business Problem Meet customer SLAs Keep applications available Don’t lose or confuse customer data Meet response time requirements Enable growth More customers More volume from existing customers New service offerings Raise bottom-line profitability Keep costs of operation as low as possible Enhance add-on revenue

Successful SaaS Systems Are Complex SaaS databases quickly evolve into “big league” topologies Tenant Data Tenant Data Shared Data Tenant Apps Tenant Apps Shared Data Tenant Analytics Tenant Analytics Tenant and Shared Data

Core Technical Issue: Multi-Tenancy Defining a flexible multi-tenant data architecture Implementing easy-to-manage clusters Scaling performance on shared databases Protecting tenant data from loss or corruption Handling failures efficiently Upgrades and maintenance windows And do it on economical open source databases!

Introducing Tungsten Clustering

What Is Tungsten? Tungsten implements master/slave clusters to: Protect data Maintain high availability Improve resource utilization Raise performance Install and set up in a few minutes Integrated backup/restore and data integrity checks Rule-driven management with automated failover Simple procedures for updates and maintenance SQL scaling for multi-tenant systems

What’s Inside Tungsten? Replication - Making copies Tungsten Replicator -- Database-neutral, platform independent master/slave replication Connectivity -- Finding databases Tungsten Connector -- Fast MySQL/PostgreSQL client to JDBC proxying Tungsten SQL Router --JDBC wrapper for high-performance and transparent failover, load-balancing, and partitioning (no proxy required) Management -- Administering the database Tungsten Manager -- Distributed administration with autonomic, rule-based configuration and no single point of failure Tungsten Monitor -- Track resource status and

Tungsten Clustering In Action Master DB Slave DB Master Host Slave Host Replicator Manager/Monitor Replicator Manager/Monitor Application Server SQL Router/Connector Application Server SQL Router/Connector Management Client Management Client

Distributed Rule-Based Management Broadcast commands and monitoring data Business Rules Manager #1 (Coordinator) Manager #3 Manager #2 Admin Client #1 Admin Client #3 Group Communications Admin Client #2 Local DBMS Resources Local DBMS Resources Local DBMS Resources

Implementing SaaS Data Management with Tungsten

Best-Practice Multi-Tenant Architecture Separate global and tenant data Global data are same for all customers Tenant data belong to individual customer Establish shards to hold tenants Bucket tenant data into databases, schemas, or groups of tables Define data services Global data service plus a set of tenant services is typical Master data in a single location for each service Services have one master and 1+N slaves for HA/scaling Add data services as you grow TIP: Use database as basic sharding unit

Typical SaaS Architecture globalds1 tenantds1 tenantds4 tenantds4 tenantds2 Global tenant definitions and billing data Per tenant data (usually many tenants per service) globalds1 Master/slave pair for each service Implementation tenantds2 tenantds4 tenantds3 tenantds1

Installing a Tungsten Cluster Four easy steps: Set up host prerequisites (MySQL, Ruby, Java) Download release from www.continuent.com and un-tar into local directory Run ‘configure’ script Ensure MySQL is correctly configured Configure cluster networking Configure replication and choose backup method Configure connectivity Perform customizations (e.g., Master VIP setup) Configure starts services automatically

Advanced But Simple Management Tools ‘ cctrl’ is the Tungsten management client List data sources in the cluster [LOGICAL] /cluster/demo/> ls Check cluster liveness [LOGICAL] /cluster/demo/> cluster heartbeat Ensure that replication is synchronized [LOGICAL] /cluster/demo/> cluster flush Start and stop operating system services [LOGICAL] /cluster/demo/> service */mysql restart Get help [LOGICAL] /cluster/demo/> help

Multiple Routes to Databases Java App Server Tungsten SQL Router PHP Application Tungsten Connector libmysqlclient.a Tungsten Cluster MySQL JDBC Driver Admin & Monitoring Admin & Monitoring Virtual IP Address

Connecting Applications to Databases Tungsten SQL Router embeds in Java applications Requires Java library substitution Very high performance, seamless planned failover Tungsten Connector proxies any MySQL application Also very high performance with seamless planned failover No library changes Master Virtual IP maintained by Tungsten Failover is not seamless (broken connections) Completely transparent to applications Direct connection to database(s) Use whatever you want You are responsible for locating master/slaves yourself

Scaling Reads on Slaves Tungsten “session-consistency” distributes tenant reads to slaves if caught up with last tenant write Works with tenants sharded by database or with independent database connections Master DB Slave DB SQL Router Record tenant write position Read if slave caught up with last tenant write

A Short Digression on Backups MySQL Server DBMS Data Files Active Host Applications Backup HOT/COLD: mysqldump Backup HOT: LVM snapshot HOT: InnoDB Hot Backup COLD: File system copy

Setting Up Backups Tungsten provides multiple backup methods Backups can run hot while DBMS is live or cold Logical backups use native DBMS utilities mysqldump, pg_dump, etc. Can run hot on slaves Slow to dump/load and in the case of MySQL may have bugs LVM (Logical Volume Manager) backups take a snapshot of the file system Very fast; run hot on slaves Tend to be large Snapshot maintenance affects write performance Script backups integrate arbitrary backup programs TIP: Configure backups and test them regularly

Running Backups With Tungsten Hot backup using one-line command on slave [LOGICAL] > datasource centos5a backup lvm Cold backups run on off-line master or slave [LOGICAL] > datasource centos5b offline [LOGICAL] > replicator centos5b offline [LOGICAL] > datasource centos5b backup [LOGICAL] > replicator centos5b online Restore with one-line command [LOGICAL] > datasource centos5a restore TIP: Keep data services small enough to backup and restore easily!

Verifying Tenant Data Copies Tungsten incremental checks verify data without stopping replication Implemented as replication events inside Tungsten Replicator Compute MD5 on all or part of tables Can do single tables or all tables in a database Run checks from manager in combination with flush command [LOGICAL] /cluster/demo/> cluster check tenant1.* [LOGICAL] /cluster/demo/> cluster flush Slaves can either fail or warn of failures TIP: Install consistency checks as a cron job TIP: Verify backups with restore + consistency check

Handling Database Failures Automatic policy mode performs master failover automatically whenever: A database crashes A replicator fails or crashes A server reboots or drops off the network Manual policy mode lets you failover manually [LOGICAL] /cluster/demo/> failover Failover procedure promotes most advanced slave And points other slaves to the slave Recover command brings back failed master [LOGICAL] /cluster/demo/> datasource centos5a recover Slaves recover without admin intervention in automatic mode

Performing Planned Maintenance Tungsten supports “daylight maintenance” To perform maintenance on a slave, just take it offline [LOGICAL] /cluster/demo/> set policy maintenance [LOGICAL] /cluster/demo/> datasource centos5b offline [LOGICAL] /cluster/demo/> replicator centos5b offline To perform maintenance on a master, first do a switch to promote a new master [LOGICAL] /cluster/demo/> switch Then perform maintenance using the slave procedure Switch the master back when you are done

Upgrading Applications and Databases If you have a maintenance window, just upgrade the master and let replication take care of it Stop old applications and replace code Run upgrade script against master Flush and wait for slave to catch up (if necessary) Restart new applications If you don’t have a maintenance window, upgrade a slave and then switch Take slave offline so that apps point to master only Upgrade the slave Turn on replication to slave so that it catches up Switch the master to the upgraded slave; take old master offline Restart applications with new code Upgrade old master and bring back online TIP: Test all upgrades in a pre-prod environment

SaaS Problems and Tungsten Solutions MySQL 5.0/5.1 Open source databases Cctrl switch operation and offline/online commands System maintenance Upgrades via replication; slave upgrade Application upgrades Automated and manual failover; live replicas; recovery procedures Handling failures Backups; live replicas; consistency checks Protecting tenant data from loss or corruption Tenant-based load balancing Performance scaling Simple installation; cctrl distributed commands Implementing easy-to-manage clusters “ Best-practices” database design Multi-tenant architecture Tungsten Features(s) Problem

Solution Roadmap for SaaS Applications Continuent is dedicated to making MySQL the foundation for SaaS businesses We will talk about these in our February 11 webinar “Secrets of Successful SaaS Data Architecture” Q2-3 Q2 Tenant management using shards Q2-3 Q2 Cross-site cluster management Q2-3 Q2 Multi-master replication Parallel replication SaaS Feature Q2-3 Q1 Prod Beta

Commercial Support for SaaS from Continuent

Tungsten Products Tungsten Enterprise offers subscription-licensed supported clustering for SaaS applications X Built-in Jump-Start consulting X Guaranteed bug fixes X 8x5, 24x7 support X Custom backup/restore X Virtual IP support X Automatic failover X X Manual failover X X SQL routing/proxying X X Advanced management tools Advanced MySQL replication Feature X X Enterprise Community

Continuent Services Practical consulting by industry experts with deep MySQL and SaaS industry background Structured Jump-Starts Hardware/OS setup, education, installation, production roll-out Solution architecture workshops Collaborative engagements to specify SaaS data management requirements, survey problems, and lay out options for solutions Proof-of-Concept implementations Fast design and implementation of specific cluster configurations

Tungsten Design Partner Program Continuent is developing advanced SaaS features in collaboration with leading SaaS vendors We offer expertise not only in database clustering but operation of SaaS applications Partner with Continuent to “push the envelope” on scalable data management using open source databases Please contact us today to find out more about joining our efforts

Conclusion SaaS applications have a complex database problem that requires “big league” architecture Continuent offers a complete solution for SaaS data management to raise overall profitability: Raise SLAs by improving availability and performance Improve management efficiency Grow customers and product offerings through scalable database architecture Increase SaaS business profitability Contact us to try Tungsten now!

Contact Information EMEA and APAC Lars Sonckin kaari 16 02600 Espoo, Finland Tel +358 50 517 9059 Fax +358 9 863 0060 Continuent Web Site: http://www.continuent.com HQ and Americas 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009 e -mail: [email_address]

How Do I Manage SaaS Databases? Tungsten clusters consist of “data sources” You can view data source state using the ‘cctrl’ command $ tungsten-manager/bin/cctrl [LOGICAL] /cluster/demo/> ls COORDINATOR[centos5a:MANUAL] ROUTERS(centos5a:10999:ONLINE) DATASOURCES: centos5a(slave:ONLINE, progress=10, latency=0.443)REPLICATOR(role=slave, master=centos5b, state=ONLINE) centos5b(master:ONLINE, progress=10)REPLICATOR(role=master, state=ONLINE) Use ‘datasource’ and ‘replicator’ commands to manage database state

How Do I Recover Failed Databases? Recover procedures depend on what actually happened to your database To fix a failed slave database, just restart the database In automatic mode, rules take care of the rest To fix a failed master, reprovision it as a slave, then add it back to the cluster [LOGICAL] /cluster/demo/> set policy maintenance [LOGICAL] /cluster/demo/> datasource centos5a slave [LOGICAL] /cluster/demo/> replicator centos5b slave [LOGICAL] /cluster/demo/> datasource centos5a welcome [LOGICAL] /cluster/demo/> set policy automatic

Continuent Tungsten - Scalable Saa S Data Management

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Continuent Tungsten - Scalable Saa S Data Management (20)

Recently uploaded (20)

Continuent Tungsten - Scalable Saa S Data Management

Editor's Notes