SlideShare a Scribd company logo

Mysql 2007 Tech At Digg V3

E
epee

Technology at digg.com initially used a single Linux server with Apache, PHP, and MySQL. It grew to use multiple servers with load balancing, master-slave replication, and Memcached for caching. Purpose-driven MySQL pools were created to isolate high database loads. While sharding was considered, it was not implemented due to issues with SQL support and increased complexity. Current challenges include physical I/O performance, optimizing queries, and managing monitoring systems.

Technology
1 of 16
Downloaded 1,551 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Technology at digg.com Elliott White III - Eli Tim Ellis - Time
What's all this then? History of Tech at Digg Standard Setup Memcached Purpose Driven MySQL Server Pools Sharding Other MySQL details
History of Tech at Digg Initially one Linux server with Apache 1.3 and PHP 4.x MySQL 4.0 and myisam tables & MySQL full-text search Started with as many open source packages as possible for rapid development. ImageMagick, Ispell, prototype/scriptaculous, etc Grew quick. One server became two. myisam become innodb. We moved to 3 servers, Apache 2.x, MySQL master-slave replication, started using memcached, moved to PHP 5.x & hired a dba And then the pace picked up yet again...
A Standard Setup Many PHP Servers behind a load balancer:
A Standard Setup Many MySQL slaves, talking to a master
A Standard Setup Randomized PHP to MySQL relations
Memcached What is it & Why use it? Why not just pregenerate pages of content? Performance gains: Caching certain chunks of data that may be used on many different pages. From that, still being able to dynamically create the page, but using cached data.
Memcached Server Farm Issues to be solved: Allowing for Failover and Redundancy Removing or ignoring stale data upon machine failure Stopping perpetuation of database slave lag Making the actual decisions of what and when to cache Solution: Write generic code that understands your data.
Purpose Driven MySQL Pools Creating separate slave pools, that are close to identical in order to isolate high database load.
Purpose Driven Pool Example
Sharding Simplest Definition: Breaking up your database into a number of smaller ones. Pros Greater performance Tweakable / Scalable Cons Loss of SQL support (JOIN) Increased PHP load Complicated programming
Possible Types of Sharding Table-based Range-based Date-based Hashed Partial Sharding
How MySQL Can Do Sharding Partition tables Federated tables MySQL Cluster (NDB) Which of these technologies does Digg use? None of them They weren't ready when we needed them
Digg MySQL Specifics Mix of MySQL 5.0 versions: 5.0.22, 5.0.27, 5.0.30, 5.0.32 A single MySQL 4.1 supporting Cacti MySQL installation & patching done via Debian apt 5.0 performs fine for us compared to 4.1 OLTP DBs are InnoDB & OLAP DB is MyISAM InnoDB backup DB for dumps and slave creation InnoDB recovery faster after slave hardware dies MyISAM great for loads and date-versioned DBs
Current DB Challenges Scaled via “buy more RAM,” can't afford that anymore Physical I/O common, so rewrite queries Convincing site architects to change features to more scalable alternatives MySQL debug binary testing Schema cruft Managing monitoring & alerting systems (Cacti) Master is a single point of failure (not as bad as it sounds) Why are our disks lying? Use diskTest.pl from: http://faemalia.net/mysqlUtils
Any Questions? For this presentation and more: http://eliw.com/ Visit http://digg.com/

More Related Content

PPTX
Using Redis as Distributed Cache for ASP.NET apps - Peter Kellner, 73rd Stre...
Redis Labs
 
PPTX
Building Scalable Web Apps - LVL.UP KL
Gareth Davies
 
PPT
MongoDB at community engine
mathraq
 
PPT
High Performance Wordpress
Gareth Davies
 
PDF
Roshan Bhattarai: Scaling WordPress for high traffic sites
wpnepal
 
PPTX
Scaling wordpress for high traffic
Roshan Bhattarai
 
PPTX
MongoDB on EC2 and EBS
Jared Rosoff
 
PPTX
Day 2 General Session Presentations RedisConf
Redis Labs
 
Using Redis as Distributed Cache for ASP.NET apps - Peter Kellner, 73rd Stre...
Redis Labs
 
Building Scalable Web Apps - LVL.UP KL
Gareth Davies
 
MongoDB at community engine
mathraq
 
High Performance Wordpress
Gareth Davies
 
Roshan Bhattarai: Scaling WordPress for high traffic sites
wpnepal
 
Scaling wordpress for high traffic
Roshan Bhattarai
 
MongoDB on EC2 and EBS
Jared Rosoff
 
Day 2 General Session Presentations RedisConf
Redis Labs
 

What's hot (16)

PPTX
Kubernetes talk at DDDSydney 2017
Hesham Amin
 
PPTX
Optimizing Your WordPress Site
ozzyr
 
PDF
Configuring MongoDB HA Replica Set on AWS EC2
ShepHertz
 
PDF
Memcache and Drupal - Vaibhav Jain
Drupal Camp Delhi
 
PPTX
Bacula4
Elsayed Atef
 
PPTX
MongoDB and Amazon Web Services: Storage Options for MongoDB Deployments
MongoDB
 
PPTX
Drupal
khinnyein chanthu
 
PDF
OpenExpo: MySQL, Where are you going?
FromDual GmbH
 
PDF
dba_lounge_Iasi: Everybody likes redis
Liviu Costea
 
PPTX
Rocking mongo db on the cloud
MongoDB
 
PDF
Growing MongoDB on AWS
colinthehowe
 
PDF
WordCamp RVA
codearachnid_test
 
PPTX
Microsoft Azure Storage Basics
Sai Kishore Naidu
 
PPTX
MySQL At Yelp
Yelp Engineering
 
PPTX
Scaling Drupal & Deployment in AWS
永对 陈
 
PDF
High Performance WordPress II
Barry Abrahamson
 
Kubernetes talk at DDDSydney 2017
Hesham Amin
 
Optimizing Your WordPress Site
ozzyr
 
Configuring MongoDB HA Replica Set on AWS EC2
ShepHertz
 
Memcache and Drupal - Vaibhav Jain
Drupal Camp Delhi
 
Bacula4
Elsayed Atef
 
MongoDB and Amazon Web Services: Storage Options for MongoDB Deployments
MongoDB
 
Drupal
khinnyein chanthu
 
OpenExpo: MySQL, Where are you going?
FromDual GmbH
 
dba_lounge_Iasi: Everybody likes redis
Liviu Costea
 
Rocking mongo db on the cloud
MongoDB
 
Growing MongoDB on AWS
colinthehowe
 
WordCamp RVA
codearachnid_test
 
Microsoft Azure Storage Basics
Sai Kishore Naidu
 
MySQL At Yelp
Yelp Engineering
 
Scaling Drupal & Deployment in AWS
永对 陈
 
High Performance WordPress II
Barry Abrahamson
 
Ad

Similar to Mysql 2007 Tech At Digg V3 (20)

PDF
MySQL for Oracle DBA -- Rocky Mountain Oracle User Group Training Days '15
Dave Stokes
 
PDF
Introductiontomysql 100420092731-phpapp01
sagaroceanic11
 
PPTX
MySQL database
lalit choudhary
 
PDF
MySQL overview
Mark Swarbrick
 
PDF
My sql crashcourse_intro_kdl
sqlhjalp
 
PDF
NOSQL Meets Relational - The MySQL Ecosystem Gains More Flexibility
Ivan Zoratti
 
PDF
🐬 The future of MySQL is Postgres 🐘
RTylerCroy
 
ODP
MySQL And Search At Craigslist
Jeremy Zawodny
 
PDF
My sql susecon_crashcourse_2012
sqlhjalp
 
PDF
Introduction to MySQL
Ted Wennmark
 
PDF
My Sql And Search At Craigslist
MySQLConference
 
PDF
The MySQL Server ecosystem in 2016
Colin Charles
 
PDF
Getting 100B Metrics to Disk
jthurman42
 
PDF
Mysql User Camp : 20-June-14 : Mysql New features and NoSQL Support
Mysql User Camp
 
PDF
Mysql User Camp : 20th June - Mysql New Features
Tarique Saleem
 
PDF
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
Alkin Tezuysal
 
PDF
MySQL overview
Marco Tusa
 
PPTX
Web scale MySQL at Facebook (Domas Mituzas)
Ontico
 
PPTX
Case Study with Answers.com on Scaling with Memcached and MySQL
answers
 
PPT
MySQL Alta Disponibilidade com Replicação
MySQL Brasil
 
MySQL for Oracle DBA -- Rocky Mountain Oracle User Group Training Days '15
Dave Stokes
 
Introductiontomysql 100420092731-phpapp01
sagaroceanic11
 
MySQL database
lalit choudhary
 
MySQL overview
Mark Swarbrick
 
My sql crashcourse_intro_kdl
sqlhjalp
 
NOSQL Meets Relational - The MySQL Ecosystem Gains More Flexibility
Ivan Zoratti
 
🐬 The future of MySQL is Postgres 🐘
RTylerCroy
 
MySQL And Search At Craigslist
Jeremy Zawodny
 
My sql susecon_crashcourse_2012
sqlhjalp
 
Introduction to MySQL
Ted Wennmark
 
My Sql And Search At Craigslist
MySQLConference
 
The MySQL Server ecosystem in 2016
Colin Charles
 
Getting 100B Metrics to Disk
jthurman42
 
Mysql User Camp : 20-June-14 : Mysql New features and NoSQL Support
Mysql User Camp
 
Mysql User Camp : 20th June - Mysql New Features
Tarique Saleem
 
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
Alkin Tezuysal
 
MySQL overview
Marco Tusa
 
Web scale MySQL at Facebook (Domas Mituzas)
Ontico
 
Case Study with Answers.com on Scaling with Memcached and MySQL
answers
 
MySQL Alta Disponibilidade com Replicação
MySQL Brasil
 
Ad

Recently uploaded (20)

PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 

Mysql 2007 Tech At Digg V3

  • 1. Technology at digg.com Elliott White III - Eli Tim Ellis - Time
  • 2. What's all this then? History of Tech at Digg Standard Setup Memcached Purpose Driven MySQL Server Pools Sharding Other MySQL details
  • 3. History of Tech at Digg Initially one Linux server with Apache 1.3 and PHP 4.x MySQL 4.0 and myisam tables & MySQL full-text search Started with as many open source packages as possible for rapid development. ImageMagick, Ispell, prototype/scriptaculous, etc Grew quick. One server became two. myisam become innodb. We moved to 3 servers, Apache 2.x, MySQL master-slave replication, started using memcached, moved to PHP 5.x & hired a dba And then the pace picked up yet again...
  • 4. A Standard Setup Many PHP Servers behind a load balancer:
  • 5. A Standard Setup Many MySQL slaves, talking to a master
  • 6. A Standard Setup Randomized PHP to MySQL relations
  • 7. Memcached What is it & Why use it? Why not just pregenerate pages of content? Performance gains: Caching certain chunks of data that may be used on many different pages. From that, still being able to dynamically create the page, but using cached data.
  • 8. Memcached Server Farm Issues to be solved: Allowing for Failover and Redundancy Removing or ignoring stale data upon machine failure Stopping perpetuation of database slave lag Making the actual decisions of what and when to cache Solution: Write generic code that understands your data.
  • 9. Purpose Driven MySQL Pools Creating separate slave pools, that are close to identical in order to isolate high database load.
  • 11. Sharding Simplest Definition: Breaking up your database into a number of smaller ones. Pros Greater performance Tweakable / Scalable Cons Loss of SQL support (JOIN) Increased PHP load Complicated programming
  • 12. Possible Types of Sharding Table-based Range-based Date-based Hashed Partial Sharding
  • 13. How MySQL Can Do Sharding Partition tables Federated tables MySQL Cluster (NDB) Which of these technologies does Digg use? None of them They weren't ready when we needed them
  • 14. Digg MySQL Specifics Mix of MySQL 5.0 versions: 5.0.22, 5.0.27, 5.0.30, 5.0.32 A single MySQL 4.1 supporting Cacti MySQL installation & patching done via Debian apt 5.0 performs fine for us compared to 4.1 OLTP DBs are InnoDB & OLAP DB is MyISAM InnoDB backup DB for dumps and slave creation InnoDB recovery faster after slave hardware dies MyISAM great for loads and date-versioned DBs
  • 15. Current DB Challenges Scaled via “buy more RAM,” can't afford that anymore Physical I/O common, so rewrite queries Convincing site architects to change features to more scalable alternatives MySQL debug binary testing Schema cruft Managing monitoring & alerting systems (Cacti) Master is a single point of failure (not as bad as it sounds) Why are our disks lying? Use diskTest.pl from: http://faemalia.net/mysqlUtils
  • 16. Any Questions? For this presentation and more: http://eliw.com/ Visit http://digg.com/