MySQL Install for Replication - Real Life Tutorial
Download as PPTX, PDF0 likes2,604 views
The document discusses the setup and optimization of a high-traffic website, tradebit.com, detailing the importance of replication, caching, and server configuration in managing large datasets and heavy traffic. It emphasizes simplicity in setup, highlights the challenges of maintaining a database under load, and suggests monitoring and fixing replication issues efficiently using tools. The author concludes that while replication can be sufficient, careful attention to configuration and code optimization is essential for maintaining performance and revenue.
![MySQL ... replication
web writes – master 1 shell writes – master 2
Dual Quad Core Dual Quad Core
32 Gig RAM, RAID5 32 Gig RAM, RAID5
Dual Quad Core Dual Quad Core Dual Quad Core Dual Quad Core
32 GB, RAID0 32 GB, RAID0 32 GB, RAID0 32 GB, RAID0
Dual Quad Core DQuad, 24 GB, Dual Quad Core DQuad, 24 GB,
32 GB, RAID0 RAID0: sphinx 32 GB, RAID0 RAID0: sphinx
Global: config.php
Config::$Dbwrite[0]=“dm1”; Config::$Dbwrite[1]=“dm2”;
Config::$Dbread[0..5] = “dsguatda.com/cmx.p1...6”;
19 © Ralf Schwoebel
next: Numbers on tb puzzler@tradebit.com / twitter: trabit](https://image.slidesharecdn.com/mysqlsetupv5final-121001160758-phpapp02/85/MySQL-Install-for-Replication-Real-Life-Tutorial-19-320.jpg)
![MySQL ... replication
Inserts, updates, etc. - September, 17th on tradebit:
• 920 new products
• 1,930 new orders
• 15,500 debug entries in the DB
• Around 35,000 additional modification operations (affiliates)
• Around 130,000 unique visits
Roughly 100,000 inserts/update operations
Roughly 3.5 Mio. read operations (hello Googlebot, Bingbot, etc.)
[root@DM1 ~]# uptime …., load average: 0.02, 0.05, 0.02
20 © Ralf Schwoebel
next: Why so much? puzzler@tradebit.com / twitter: trabit](https://image.slidesharecdn.com/mysqlsetupv5final-121001160758-phpapp02/85/MySQL-Install-for-Replication-Real-Life-Tutorial-20-320.jpg)
![MySQL ... replication
static function dbUpdateRowsById($strTable, $strSetCols, $strIdName, $strSelectIdSql) {
$nMax = 1000;
$nAff=0;
$result=dbQuery($strSelectIdSql);
if ($result) {
$strIds = '';
$i=0;
while (($row=mysql_fetch_array($result))) {
$strIds .= ($strIds?',':'').$row[0];
$i++;
if ($i>$nMax) {
dbQuery("update $strTable set $strSetCols where $strIdName in ($strIds)");
if (mysql_errno()) return false;
$nAff += mysql_affected_rows();
$i=0;
$strIds='';
}
}
dbFreeResult($result);
if ($strIds) {
dbQuery("update $strTable set $strSetCols where $strIdName in ($strIds)");
$nAff += mysql_affected_rows();
}
}
return $nAff;
}
24 © Ralf Schwoebel
next: Debug line puzzler@tradebit.com / twitter: trabit](https://image.slidesharecdn.com/mysqlsetupv5final-121001160758-phpapp02/85/MySQL-Install-for-Replication-Real-Life-Tutorial-24-320.jpg)
![MySQL ... Replication Monitoring
Simple checking:
• Count the crucial entries via script
• grep the important infos
[root@w1 ~]# cat /usr/bin/wdbsynccheck
#!/bin/bash
echo "___________________________________________________________ DM1 Master: |"
ssh dm1 'mysql -e "show master statusG"' |egrep "(File|Position)"
ssh dm1 'mysql tradebit -e "select count(TB_INDEX) AS SALESCOUNT from TBSALES;"‘
echo "___________________________________________________________ DM2 Slave: |"
ssh dm2 'mysql -e "show slave statusG"'
|egrep "(Master_Log|Seconds_Behind_Master|Slave_IO_Running|Slave_SQL_Running)“
echo "___________________________________________________________ DS1 Slave: |"
ssh ds1 'mysql -e "show slave statusG"'
|egrep "(Master_Log|Seconds_Behind_Master|Slave_IO_Running|Slave_SQL_Running)"
26 © Ralf Schwoebel
next: monitor with PT puzzler@tradebit.com / twitter: trabit](https://image.slidesharecdn.com/mysqlsetupv5final-121001160758-phpapp02/85/MySQL-Install-for-Replication-Real-Life-Tutorial-26-320.jpg)
![MySQL ... Replication Fixing
Poor admin solution…
[root@DS7-SP5 ~]# more /cloud/MySQLbackup.sh
#!/bin/bash
cd /cloud/baks/
mysql MYtbDB -e "show slave statusG" >masterstatus_tradebit.txt
for i in `mysql MYtbDB -e "show tables" | grep -v Tables_`
do mysqldump --add-drop-table --complete-insert
--insert-ignore MYtbDB $i >tb_$i.sql
done
=> Dumps around 80 Gig of text in roughly 15 minutes
29 © Ralf Schwoebel
next: Poor man fixing 2 puzzler@tradebit.com / twitter: trabit](https://image.slidesharecdn.com/mysqlsetupv5final-121001160758-phpapp02/85/MySQL-Install-for-Replication-Real-Life-Tutorial-29-320.jpg)
MySQL Install for Replication - Real Life Tutorial
- 1. Sometimes replication is enough! 1 © Ralf Schwoebel next: Content puzzler@tradebit.com / twitter: trabit
- 2. Content • Intro • A real life setup • Replication with MySQL 5.5 • Load balancing tricks • Sphinx, we love it! • The crazy daily • Q&A 2 © Ralf Schwoebel next: Intro puzzler@tradebit.com / twitter: trabit
- 3. Intro: Tradebit and Ralf Ralf (puzzler) – born 1973: • BBS and Game Dev on Commodore Amiga • Started online shops in ‘95 - Perl on SuSE • Codes in PHP, RealBasic, Scripts in bash • Went into SEO in 2002 Tradebit.com – born 2004: • First a simple download shop • Became a marketplace • Grew from single server to mini-cloud • Turns around $5 Mio. per year 3 © Ralf Schwoebel next: Simplicity puzzler@tradebit.com / twitter: trabit
- 4. What we will talk about... Source: http://www.getelastic.com/site-speed-infographic/ 4 © Ralf Schwoebel next: More Revenue puzzler@tradebit.com / twitter: trabit
- 5. First – What do you REALLY need? 5 © Ralf Schwoebel next: The Bottleneck puzzler@tradebit.com / twitter: trabit
- 6. The bottleneck is your setup! 6 © Ralf Schwoebel next: The whole setup puzzler@tradebit.com / twitter: trabit
- 7. Real Life Setup: Tradebit.com 2012 USA Latin A. Europe Asia Australia Content Distribution Network: cdn.tradebit.org (Softlayer) pa pb pc pd pe pf pg Squid Layer, FTP Frontend, SNORT & Guardian (iptables) w1 w2 w3 w4 w5 w6 PHP Layer, Apache 2, Local Memcache, apc dm1 dm2 ds1 ds2 ds3 ds4 ds5 ds6 Database Layer: 2 Masters, 6 Slaves s1 s2 s3 s4 cloudfuse sp1 sp2 Storage Layer: NFS Exports, 60 Terabytes MD3k, 500 TB cloud Search Layer: Sphinx 7 © Ralf Schwoebel next: Does it make sense? puzzler@tradebit.com / twitter: trabit
- 8. Setup: Does it make sense? Oh, yeah – it does! 8 © Ralf Schwoebel next: squid puzzler@tradebit.com / twitter: trabit
- 9. Great things about ... squid Squid & Snort Layer: • Takes the load from the webservers (and DB servers) • Makes you IPv6 ready in less than 20 minutes • Keeps the basic hack attacks under control • Less vulnerable on hardware failure (just with Round Robin) 9 © Ralf Schwoebel next: Caching in general puzzler@tradebit.com / twitter: trabit
- 10. Great things about ... Caching APC: One of 6 10 © Ralf Schwoebel next: Other tools puzzler@tradebit.com / twitter: trabit
- 11. Great things about ... The Rest Memcache: Holds most requested pages as a whole NFS: Darn stable, stone old, no surprises Sphinx: Stable, fast and easy to manage Hosted CDN: Optional quick solution for your static files 11 © Ralf Schwoebel next: Bad stuff puzzler@tradebit.com / twitter: trabit
- 12. Some bad things... Things you probably do not want to do in this setup: • directory browsing via NFS for anything speedy • have 3rd party software open holes (aka Wordpress) • having the SNORT behave oddly (like blocking Google) • loosing orders (who wants that in any setup?) 12 © Ralf Schwoebel next: Live Demo puzzler@tradebit.com / twitter: trabit
- 13. Short Live Demo http://www.tradebit.com/filedetail.php/178346105-new-york-empire-state-of-mind- 13 © Ralf Schwoebel next: Replication basics puzzler@tradebit.com / twitter: trabit
- 14. MySQL ... Replication basics Replication means: write to a MASTER MySQL replicates to a SLAVE 14 © Ralf Schwoebel next: When is replication ok? puzzler@tradebit.com / twitter: trabit
- 15. MySQL ... Replication basics Replication is ONLY SOMETIMES enough: • Consumer websites with heavy traffic reading (4 Mio UV/day is OK, I vouch for that...) • Large datasets for archiving • NON critical requests: like statistics or backup strategies For A LOT of write events throughout the day (think constantly 10 million or more a day): you would need a different approach! 15 © Ralf Schwoebel next: Replication and friends puzzler@tradebit.com / twitter: trabit
- 16. MySQL ... and friends MySQL Setup for millions of products: • Is good, but you need to know what you are doing! • Search: forget “match against” – quickly! • Browsing: any filtering on big tables is too slow • 2 rules (I learned from Percona): • Use numbers/boolean where possible • Avoid “like ‘%xyz%’” like the plague! 16 © Ralf Schwoebel next: DB sizes puzzler@tradebit.com / twitter: trabit
- 17. MySQL ... and friends Main product table: • 37 million products • Around 12 million in 2 or more languages 17 © Ralf Schwoebel next: Config tricks puzzler@tradebit.com / twitter: trabit
- 18. MySQL ... replication On a side note: InnoDB and some config basics! innodb_file_per_table replicate-ignore-db=mysql replicate-ignore-db=test replicate-wild-ignore-table=tradebitTmp.% slave-skip-errors=all 18 © Ralf Schwoebel next: Replication servers puzzler@tradebit.com / twitter: trabit
- 19. MySQL ... replication web writes – master 1 shell writes – master 2 Dual Quad Core Dual Quad Core 32 Gig RAM, RAID5 32 Gig RAM, RAID5 Dual Quad Core Dual Quad Core Dual Quad Core Dual Quad Core 32 GB, RAID0 32 GB, RAID0 32 GB, RAID0 32 GB, RAID0 Dual Quad Core DQuad, 24 GB, Dual Quad Core DQuad, 24 GB, 32 GB, RAID0 RAID0: sphinx 32 GB, RAID0 RAID0: sphinx Global: config.php Config::$Dbwrite[0]=“dm1”; Config::$Dbwrite[1]=“dm2”; Config::$Dbread[0..5] = “dsguatda.com/cmx.p1...6”; 19 © Ralf Schwoebel next: Numbers on tb puzzler@tradebit.com / twitter: trabit
- 20. MySQL ... replication Inserts, updates, etc. - September, 17th on tradebit: • 920 new products • 1,930 new orders • 15,500 debug entries in the DB • Around 35,000 additional modification operations (affiliates) • Around 130,000 unique visits Roughly 100,000 inserts/update operations Roughly 3.5 Mio. read operations (hello Googlebot, Bingbot, etc.) [root@DM1 ~]# uptime …., load average: 0.02, 0.05, 0.02 20 © Ralf Schwoebel next: Why so much? puzzler@tradebit.com / twitter: trabit
- 21. MySQL ... replication So, 3.5 Mio reads per day, 100k writing… fine, but: • Why 8 servers? 2 x 4 combination gives us failover! The site can fully run (slower) with 1 strain… • Why no cluster? Replication is MUCH easier to setup than a cluster and recovery is easier! 21 © Ralf Schwoebel next: When to write? puzzler@tradebit.com / twitter: trabit
- 22. MySQL ... replication 22 © Ralf Schwoebel next: SQL principles on updates puzzler@tradebit.com / twitter: trabit
- 23. MySQL ... replication Be careful, what you update: UPDATE mytable SET mycol=2 WHERE my2ndcol LIKE ‘%.jpg’ is EVIL and could easily kill your replication setup on large tables! 23 © Ralf Schwoebel next: PHP code for updates puzzler@tradebit.com / twitter: trabit
- 24. MySQL ... replication static function dbUpdateRowsById($strTable, $strSetCols, $strIdName, $strSelectIdSql) { $nMax = 1000; $nAff=0; $result=dbQuery($strSelectIdSql); if ($result) { $strIds = ''; $i=0; while (($row=mysql_fetch_array($result))) { $strIds .= ($strIds?',':'').$row[0]; $i++; if ($i>$nMax) { dbQuery("update $strTable set $strSetCols where $strIdName in ($strIds)"); if (mysql_errno()) return false; $nAff += mysql_affected_rows(); $i=0; $strIds=''; } } dbFreeResult($result); if ($strIds) { dbQuery("update $strTable set $strSetCols where $strIdName in ($strIds)"); $nAff += mysql_affected_rows(); } } return $nAff; } 24 © Ralf Schwoebel next: Debug line puzzler@tradebit.com / twitter: trabit
- 25. MySQL ... replication Never stop to optimize… 25 © Ralf Schwoebel next: Monitor puzzler@tradebit.com / twitter: trabit
- 26. MySQL ... Replication Monitoring Simple checking: • Count the crucial entries via script • grep the important infos [root@w1 ~]# cat /usr/bin/wdbsynccheck #!/bin/bash echo "___________________________________________________________ DM1 Master: |" ssh dm1 'mysql -e "show master statusG"' |egrep "(File|Position)" ssh dm1 'mysql tradebit -e "select count(TB_INDEX) AS SALESCOUNT from TBSALES;"‘ echo "___________________________________________________________ DM2 Slave: |" ssh dm2 'mysql -e "show slave statusG"' |egrep "(Master_Log|Seconds_Behind_Master|Slave_IO_Running|Slave_SQL_Running)“ echo "___________________________________________________________ DS1 Slave: |" ssh ds1 'mysql -e "show slave statusG"' |egrep "(Master_Log|Seconds_Behind_Master|Slave_IO_Running|Slave_SQL_Running)" 26 © Ralf Schwoebel next: monitor with PT puzzler@tradebit.com / twitter: trabit
- 27. MySQL ... Replication Monitoring Or use the percona tools… http://www.percona.com/doc/percona-toolkit/2.1/pt-heartbeat.html 27 © Ralf Schwoebel next: It will fail puzzler@tradebit.com / twitter: trabit
- 28. MySQL ... When *it hits the fan... Replication gets out of sync, sooner or later: 28 © Ralf Schwoebel next: Poor man fixing puzzler@tradebit.com / twitter: trabit
- 29. MySQL ... Replication Fixing Poor admin solution… [root@DS7-SP5 ~]# more /cloud/MySQLbackup.sh #!/bin/bash cd /cloud/baks/ mysql MYtbDB -e "show slave statusG" >masterstatus_tradebit.txt for i in `mysql MYtbDB -e "show tables" | grep -v Tables_` do mysqldump --add-drop-table --complete-insert --insert-ignore MYtbDB $i >tb_$i.sql done => Dumps around 80 Gig of text in roughly 15 minutes 29 © Ralf Schwoebel next: Poor man fixing 2 puzzler@tradebit.com / twitter: trabit
- 30. MySQL ... Replication Fixing Poor admin solution… play it back in stop slave; mysql DBname <table.sql mysql> CHANGE MASTER TO -> MASTER_HOST=’dm1', -> MASTER_USER=‚*', -> MASTER_PASSWORD=‘*', -> MASTER_LOG_FILE=' mylog.000416 ', // Master_Log_File -> MASTER_LOG_POS=12345; // Read_Master_Log_Pos 30 © Ralf Schwoebel next: PT fixing puzzler@tradebit.com / twitter: trabit
- 31. MySQL ... Replication Fixing Pro solution: Percona tools rule, thanks buddies! pt-table-sync --print u=tradebit,h=dm2,D=tradebit,t=$t $i --password * --where $wh >"$d/$i-$t“ with example values: pt-table-sync --print u=tradebit,h=dm2,D=tradebit,t=TBSALES ds5 --password * --where “TB_INDEX>100” >fix.sql (note: --sync-to-master) 31 © Ralf Schwoebel next: Summary puzzler@tradebit.com / twitter: trabit
- 32. MySQL ... Replication Summary Conclusion: • Simplicity – do not let people talk you into complicated setups • Speed – rather kill features than risk revenue • Relativity – your config and code must co-exist optimized • Use Tools – think outside the box to relieve stress Yes, replication CAN be enough. 32 © Ralf Schwoebel next: Downloads puzzler@tradebit.com / twitter: trabit
- 33. Thanks I am available for questions: Ralf Schwoebel eMail: puzzler@tradebit.com Twitter: trabit XING: https://www.xing.com/profile/Ralf_Schwoebel Download: http://www.slideshare.net/extremelongusername 33 © Ralf Schwoebel puzzler@tradebit.com / twitter: trabit