SlideShare a Scribd company logo

Scalability at GROU.PS

Download as PPTX, PDF
E
esokullu

The document outlines the scalability challenges and technological architecture of grou.ps, which includes a distributed system with over 25 servers and various Amazon technologies. It discusses performance optimization techniques, such as MySQL replication, caching with Memcache, and sysctl tunings for handling high traffic. Additionally, it emphasizes the importance of monitoring and database logging solutions, as well as ongoing improvements to the system.

Technology
1 of 27
Downloaded 50 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Scalability at GROU.PSEmreSokullu
DisclaimerWe’re not fully there yetWe hire: jobs@groups-inc.com
Challenges @ GROU.PS3M unique visitors per month120M page views1PB assets to be served every monthVideo,Photos, FilesSupport for 5Gbit/sVery dynamic pages:With social networks; p(u,t) = HTMLp(g,u,t) = HTML -> WHERE group_id = ? AND …
What is GROU.PS ?
Scalability at GROU.PS
Scalability at GROU.PS
Scalability at GROU.PS
Scalability at GROU.PS
Distributed Architecture25+ servers, S3 cloud, EdgeCast CDN4 cores + All Linux: Red HatSome Debian, Ubuntu, CentOS
Amazon TechnologiesS3CloudFrontEC2 (elastic IP and persistent storage)SimpleDBQueue technologies, distributed hadoop and more…
Amazon TechnologiesDownside: Not so cheapBad database performance
Serving Content?Use MogileFSDistributed file servingUse CDNhot content served off from local serversSysctl tunings needed!
Our typical sysctl additionsnet.ipv4.tcp_syncookies = 1net.ipv4.tcp_synack_retries = 2## Emre edited# http://www.oracle-base.com/articles/11g/OracleDB11gR1InstallationOnFedora8.phpkernel.shmall = 2097152kernel.shmmax = 2147483648kernel.shmmni = 4096# semaphores: semmsl, semmns, semopm, semmnikernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default=4194304#net.core.rmem_max=4194304net.core.wmem_default=262144#net.core.wmem_max=262144fs.file-max=5049800vm.swappiness=10## Emre edited# from http://forums.softlayer.com/showthread.php?t=3252net.ipv4.tcp_rmem = 4096 87380 8388608net.ipv4.tcp_wmem = 4096 87380 8388608net.core.rmem_max = 8388608net.core.wmem_max = 8388608net.core.netdev_max_backlog = 5000net.ipv4.tcp_window_scaling = 1net.ipv4.ip_nonlocal_bind=1# http://rackerhacker.com/2007/08/24/apache-no-space-left-on-device-couldnt-create-accept-lock/kernel.msgmni = 1024kernel.sem = 250 256000 32 1024net.ipv4.ip_conntrack_max = 524288net.ipv4.netfilter.ip_conntrack_max = 524288
MySQLLoad off via memcache$memcache->set(“group_by_name.jtpd”, 1122, false, 0);$memcache->set(“home_module_html.1122”,…, true, 30);function getGroupID($group_name) { global $memcache; if( !isset($memcache) || ($res=($memcache->get(“group_by_name.{$group_name}”)))===false ) { // get it from mysql and memcache } else { return $res; // serve from memcache }}
MySQLReplication easySplit ReadsWhat about writes?That’s where sharding comes to playVertical ShardingHorizontal ShardingMMM
MySQLRuns poorly on multi-coresquery_cache_size = 0 # on masterquery_cache_type = 0 # on masterthread_concurrency = 8 # total coresmax_connections = 750 # shouldn’t exceed thatinnodb_buffer_pool_size = 10G # a little less than the total amount
MySQL Query OptimizationINDEX group, userWHERE group = ? AND user = ?Not WHERE user = ? AND group = ?B-tree
MySQL Query OptimizationSHOW PROCESSLISTMaatkit, mk-query-digestPercona builds
NOSQLVoldemort, LinkedinCassandra, FacebookTokyo Cabinet, mixi
LoggingDatabase logging is not the solutionFile system is expensive tooA legal necessity
LoggingSolution:Scribe & ThriftBy FacebookEventually consistent
Nginx & libevent
Nginx & libeventHandles 10000 connections5gbit/sRamblerWordpressGrou.ps
PostfixRun multiple instancesSpam Clusters
MonitoringMunin + monitOther alternatives:CactiNagiosHyperic – vmware
PHP
More to come on my bloghttp://emresokullu.comMore fine tuning tipsBecome a member of my communityLove grou.ps ;)Convert to PHPWe’re hiring: jobs@groups-inc.com

More Related Content

PDF
Hosting MongoDB & our experience with database as a service
Grzegorz Gwoźdź
 
ODP
Hosting Drupal on Amazon EC2
Kornel Lugosi
 
PPTX
Drupal in the cloud with Windows Azure
LeTesteur
 
PDF
Bacd zenoss
ke4qqq
 
KEY
CHI - YAPC NA 2012
jonswar
 
ZIP
CHI-YAPC-2009
jonswar
 
PPTX
Heap Dump Analysis - AEM: Real World Issues
Kanika Gera
 
ODP
Itb session v_memcached
Skills Matter
 
Hosting MongoDB & our experience with database as a service
Grzegorz Gwoźdź
 
Hosting Drupal on Amazon EC2
Kornel Lugosi
 
Drupal in the cloud with Windows Azure
LeTesteur
 
Bacd zenoss
ke4qqq
 
CHI - YAPC NA 2012
jonswar
 
CHI-YAPC-2009
jonswar
 
Heap Dump Analysis - AEM: Real World Issues
Kanika Gera
 
Itb session v_memcached
Skills Matter
 

What's hot (19)

PPTX
Kenshoo - Use Hadoop, One Week, No Coding
MapR Technologies
 
PPTX
Azure Recovery Services
Pavel Revenkov
 
PPTX
Cloud init and cloud provisioning [openstack summit vancouver]
Joshua Harlow
 
PPTX
Migrating Existing Open Source Machine Learning to Azure
Revolution Analytics
 
PDF
Sofia WP User Group Presentation
Daniel Kanchev
 
PDF
High Performance Drupal
Chapter Three
 
PPT
Day of Cloud: Amazon EC2
cmcavoy
 
PPTX
Running JVM in Docker
Uri Savelchev
 
PPT
Implementing High Performance Drupal Sites
Shri Kumar
 
PDF
Rich storytelling with Drupal, Paragraphs and Islandora DAMS
alxbrdg
 
PDF
Scaling in Mind (Case study of Drupal Core)
jimyhuang
 
PPTX
Automating with Ansible
Ricardo Schmidt
 
PDF
Migratory Workloads Across Clouds with Nomad
Philip Watts
 
PDF
Environment for training models
FlyElephant
 
PDF
Practical Hadoop Big Data Training Course by Certified Architect
Kamal A
 
PPTX
Performance Demystified for SQL Server on Azure Virtual Machines
Amit Banerjee
 
PDF
Optimize drupal
Le Thanh Sang
 
PDF
CloudOps CloudStack Days, Austin April 2015
CloudOps2005
 
PDF
Lessons Learned at Wakoopa
Menno van der Sman
 
Kenshoo - Use Hadoop, One Week, No Coding
MapR Technologies
 
Azure Recovery Services
Pavel Revenkov
 
Cloud init and cloud provisioning [openstack summit vancouver]
Joshua Harlow
 
Migrating Existing Open Source Machine Learning to Azure
Revolution Analytics
 
Sofia WP User Group Presentation
Daniel Kanchev
 
High Performance Drupal
Chapter Three
 
Day of Cloud: Amazon EC2
cmcavoy
 
Running JVM in Docker
Uri Savelchev
 
Implementing High Performance Drupal Sites
Shri Kumar
 
Rich storytelling with Drupal, Paragraphs and Islandora DAMS
alxbrdg
 
Scaling in Mind (Case study of Drupal Core)
jimyhuang
 
Automating with Ansible
Ricardo Schmidt
 
Migratory Workloads Across Clouds with Nomad
Philip Watts
 
Environment for training models
FlyElephant
 
Practical Hadoop Big Data Training Course by Certified Architect
Kamal A
 
Performance Demystified for SQL Server on Azure Virtual Machines
Amit Banerjee
 
Optimize drupal
Le Thanh Sang
 
CloudOps CloudStack Days, Austin April 2015
CloudOps2005
 
Lessons Learned at Wakoopa
Menno van der Sman
 
Ad

Viewers also liked (17)

PDF
Mayor Emanuel meets with Responsible Retailers group to improve liquor store ...
Jaqpott
 
PPT
E-Tohum Girisimcilik Sunumu
esokullu
 
PDF
Violence Stops Here2011 Tearsheet
Jaqpott
 
PPSX
2011 Violence Stops Here Campaign Details Finalanalysis San
Jaqpott
 
PPT
Project Proteus Proteus Overview Q404
Jaqpott
 
PPT
Value Stream Mapping
Ubersoldat
 
PDF
Project management in a visual way
Sophie Uesson
 
PPTX
Facebook Connect & Facebook Platform In Use At GROU.PS
esokullu
 
PPT
Five Whys
Ubersoldat
 
PPT
Silikon vadisi'nde turk olmak 2014 odtu
esokullu
 
PPT
Quality Circle
Ubersoldat
 
PPT
The Basics of Creating a Metric
Ubersoldat
 
PPT
Waste Overview and Benefits
Ubersoldat
 
PPT
Reunion 1º trrm 13 14
jcromgar
 
PPT
Root Cause Corrective Action
Ubersoldat
 
PPT
Leading High Performance Teams
Ubersoldat
 
PPT
Conducting Effective Meetings
Ubersoldat
 
Mayor Emanuel meets with Responsible Retailers group to improve liquor store ...
Jaqpott
 
E-Tohum Girisimcilik Sunumu
esokullu
 
Violence Stops Here2011 Tearsheet
Jaqpott
 
2011 Violence Stops Here Campaign Details Finalanalysis San
Jaqpott
 
Project Proteus Proteus Overview Q404
Jaqpott
 
Value Stream Mapping
Ubersoldat
 
Project management in a visual way
Sophie Uesson
 
Facebook Connect & Facebook Platform In Use At GROU.PS
esokullu
 
Five Whys
Ubersoldat
 
Silikon vadisi'nde turk olmak 2014 odtu
esokullu
 
Quality Circle
Ubersoldat
 
The Basics of Creating a Metric
Ubersoldat
 
Waste Overview and Benefits
Ubersoldat
 
Reunion 1º trrm 13 14
jcromgar
 
Root Cause Corrective Action
Ubersoldat
 
Leading High Performance Teams
Ubersoldat
 
Conducting Effective Meetings
Ubersoldat
 
Ad

Similar to Scalability at GROU.PS (20)

PPTX
EEDC 2010. Scaling Web Applications
Expertos en TI
 
PDF
Scaling MySQL -- Swanseacon.co.uk
Dave Stokes
 
PDF
MySQL Shell/AdminAPI - MySQL Architectures Made Easy For All!
Miguel Araújo
 
PPS
Scalable Web Architectures - Common Patterns & Approaches
Cal Henderson
 
PPS
Scalable Web Arch
royans
 
PPS
Web20expo Scalable Web Arch
royans
 
PPS
Web20expo Scalable Web Arch
mclee
 
PPS
Web20expo Scalable Web Arch
guest18a0f1
 
PPT
Gavin M
Ontico
 
PPS
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Cal Henderson
 
PPTX
Infrastructure review - Shining a light on the Black Box
Miklos Szel
 
PDF
My Sql And Search At Craigslist
MySQLConference
 
PPTX
Black Friday and Cyber Monday- Best Practices for Your E-Commerce Database
Tim Vaillancourt
 
ODP
MNPHP Scalable Architecture 101 - Feb 3 2011
Mike Willbanks
 
PDF
Memcached Code Camp 2009
NorthScale
 
PDF
Your backend architecture is what matters slideshare
Colin Charles
 
PDF
Deployment
rogerbodamer
 
PDF
System Design.pdf
JitendraYadav351971
 
PDF
System design handwritten notes guidance
Shabista Imam
 
ODP
Scalable Architecture 101
Mike Willbanks
 
EEDC 2010. Scaling Web Applications
Expertos en TI
 
Scaling MySQL -- Swanseacon.co.uk
Dave Stokes
 
MySQL Shell/AdminAPI - MySQL Architectures Made Easy For All!
Miguel Araújo
 
Scalable Web Architectures - Common Patterns & Approaches
Cal Henderson
 
Scalable Web Arch
royans
 
Web20expo Scalable Web Arch
royans
 
Web20expo Scalable Web Arch
mclee
 
Web20expo Scalable Web Arch
guest18a0f1
 
Gavin M
Ontico
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Cal Henderson
 
Infrastructure review - Shining a light on the Black Box
Miklos Szel
 
My Sql And Search At Craigslist
MySQLConference
 
Black Friday and Cyber Monday- Best Practices for Your E-Commerce Database
Tim Vaillancourt
 
MNPHP Scalable Architecture 101 - Feb 3 2011
Mike Willbanks
 
Memcached Code Camp 2009
NorthScale
 
Your backend architecture is what matters slideshare
Colin Charles
 
Deployment
rogerbodamer
 
System Design.pdf
JitendraYadav351971
 
System design handwritten notes guidance
Shabista Imam
 
Scalable Architecture 101
Mike Willbanks
 

Recently uploaded (20)

PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PPTX
A Presentation on Artificial Intelligence
memembruh336
 
PDF
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
PDF
Transform Your ITIL® 4 & ITSM Strategy with AI in 2025.pdf
PDCA Consulting
 
PDF
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
PPTX
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 
PDF
WOOl fibre morphology and structure.pdf for textiles
Rajendrakumar868651
 
PPTX
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...
bonakiduza1024
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Mushroom cultivation and it's methods.pdf
mailmeonrishi
 
PDF
1 - Historical Antecedents, Social Consideration.pdf
tuazon2030627
 
PDF
Web App vs Mobile App What Should You Build First.pdf
KodekX
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PPTX
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
rameswartudu05
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
A Presentation on Artificial Intelligence
memembruh336
 
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
Transform Your ITIL® 4 & ITSM Strategy with AI in 2025.pdf
PDCA Consulting
 
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 
WOOl fibre morphology and structure.pdf for textiles
Rajendrakumar868651
 
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...
bonakiduza1024
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Mushroom cultivation and it's methods.pdf
mailmeonrishi
 
1 - Historical Antecedents, Social Consideration.pdf
tuazon2030627
 
Web App vs Mobile App What Should You Build First.pdf
KodekX
 
Encapsulation theory and applications.pdf
gurumoop
 
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
rameswartudu05
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 

Scalability at GROU.PS

  • 2. DisclaimerWe’re not fully there yetWe hire: jobs@groups-inc.com
  • 3. Challenges @ GROU.PS3M unique visitors per month120M page views1PB assets to be served every monthVideo,Photos, FilesSupport for 5Gbit/sVery dynamic pages:With social networks; p(u,t) = HTMLp(g,u,t) = HTML -> WHERE group_id = ? AND …
  • 9. Distributed Architecture25+ servers, S3 cloud, EdgeCast CDN4 cores + All Linux: Red HatSome Debian, Ubuntu, CentOS
  • 10. Amazon TechnologiesS3CloudFrontEC2 (elastic IP and persistent storage)SimpleDBQueue technologies, distributed hadoop and more…
  • 11. Amazon TechnologiesDownside: Not so cheapBad database performance
  • 12. Serving Content?Use MogileFSDistributed file servingUse CDNhot content served off from local serversSysctl tunings needed!
  • 13. Our typical sysctl additionsnet.ipv4.tcp_syncookies = 1net.ipv4.tcp_synack_retries = 2## Emre edited# http://www.oracle-base.com/articles/11g/OracleDB11gR1InstallationOnFedora8.phpkernel.shmall = 2097152kernel.shmmax = 2147483648kernel.shmmni = 4096# semaphores: semmsl, semmns, semopm, semmnikernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default=4194304#net.core.rmem_max=4194304net.core.wmem_default=262144#net.core.wmem_max=262144fs.file-max=5049800vm.swappiness=10## Emre edited# from http://forums.softlayer.com/showthread.php?t=3252net.ipv4.tcp_rmem = 4096 87380 8388608net.ipv4.tcp_wmem = 4096 87380 8388608net.core.rmem_max = 8388608net.core.wmem_max = 8388608net.core.netdev_max_backlog = 5000net.ipv4.tcp_window_scaling = 1net.ipv4.ip_nonlocal_bind=1# http://rackerhacker.com/2007/08/24/apache-no-space-left-on-device-couldnt-create-accept-lock/kernel.msgmni = 1024kernel.sem = 250 256000 32 1024net.ipv4.ip_conntrack_max = 524288net.ipv4.netfilter.ip_conntrack_max = 524288
  • 14. MySQLLoad off via memcache$memcache->set(“group_by_name.jtpd”, 1122, false, 0);$memcache->set(“home_module_html.1122”,…, true, 30);function getGroupID($group_name) { global $memcache; if( !isset($memcache) || ($res=($memcache->get(“group_by_name.{$group_name}”)))===false ) { // get it from mysql and memcache } else { return $res; // serve from memcache }}
  • 15. MySQLReplication easySplit ReadsWhat about writes?That’s where sharding comes to playVertical ShardingHorizontal ShardingMMM
  • 16. MySQLRuns poorly on multi-coresquery_cache_size = 0 # on masterquery_cache_type = 0 # on masterthread_concurrency = 8 # total coresmax_connections = 750 # shouldn’t exceed thatinnodb_buffer_pool_size = 10G # a little less than the total amount
  • 17. MySQL Query OptimizationINDEX group, userWHERE group = ? AND user = ?Not WHERE user = ? AND group = ?B-tree
  • 18. MySQL Query OptimizationSHOW PROCESSLISTMaatkit, mk-query-digestPercona builds
  • 20. LoggingDatabase logging is not the solutionFile system is expensive tooA legal necessity
  • 21. LoggingSolution:Scribe & ThriftBy FacebookEventually consistent
  • 23. Nginx & libeventHandles 10000 connections5gbit/sRamblerWordpressGrou.ps
  • 25. MonitoringMunin + monitOther alternatives:CactiNagiosHyperic – vmware
  • 26. PHP
  • 27. More to come on my bloghttp://emresokullu.comMore fine tuning tipsBecome a member of my communityLove grou.ps ;)Convert to PHPWe’re hiring: jobs@groups-inc.com