Abstract image of a woman sitting on books and studying on a laptop

Scaling Pinterest

C4Media, profile picture
C4Media

The document outlines the evolution and scaling strategies employed by Pinterest from its inception in 2010 through various technology deployments and growth phases. It details the technical infrastructure choices, including the use of Amazon Web Services, Python, and MySQL, as well as challenges experienced during scalability efforts. The presentation emphasizes lessons learned, future goals, and seeks to refine operational methodologies for improved performance and user experience.

TechnologyDesign
Related topics:
architecture-design
1 of 64
1
2
Most read
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Most read
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
Scaling Pinterest Marty Weiner Cloud Ninja Yash Nelapati Ascii Artist Monday, November 11, 13
Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /scaling-pinterest InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
Presented at QCon San Francisco www.qconsf.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
Evolution Scaling Pinterest Monday, November 11, 13
Growth March 2010 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
Growth March 2010 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
Growth March 2010 Page views per day · · · · RackSpace 1 small Web Engine 1 small MySQL DB 1 Engineer + 2 Founders Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
Growth March 2010 Scaling Pinterest Monday, November 11, 13
Growth March 2010 Scaling Pinterest Monday, November 11, 13
Growth January 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012
Growth January 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012
Growth January 2011 Page views per day · Amazon EC2 + S3 + CloudFront · · · 1 NGinX, 4 Web Engines 1 MySQL DB + 1 Read Slave 1 Task Queue + 2 Task Processors · · 1 MongoDB 2 Engineers + 2 Founders Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012
Scaling Pinterest Monday, November 11, 13
Growth September 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
Growth September 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
Growth September 2011 Page views per day · · Amazon EC2 + S3 + CloudFront 2 NGinX, 16 Web Engines + 2 API Engines · 5 Functionally Sharded MySQL DB + 9 read slaves · · 4 Cassandra Nodes 15 Membase Nodes (3 separate clusters) · · · · · · 8 Memcache Nodes 10 Redis Nodes 3 Task Routers + 4 Task Processors 4 Elastic Search Nodes 3 Mongo Clusters 3 Engineers (8 Total) Scaling Pinterest Monday, November 11, 13 Mar 2010 Jan 2011 Jan 2012 May 2012
It will fail. Keep it simple. Scaling Pinterest Monday, November 11, 13
If you’re the biggest user of a technology, the challenges will be greatly amplified Scaling Pinterest Monday, November 11, 13
Growth January 2012 Scaling Pinterest Monday, November 11, 13
Growth April 2012 Page views per day Mar 2010 Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
Growth April 2012 Page views per day Mar 2010 Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
Growth April 2012 Page views per day · · · · Amazon EC2 + S3 + Edge Cast 135 Web Engines + 75 API Engines 10 Service Instances 80 MySQL DBs (m1.xlarge) + 1 slave each · · · 110 Redis Instances 60 Memcache Instances 2 Redis Task Manager + 60 Task Mar 2010 Processors · 3rd party sharded Solr Scaling Pinterest Monday, November 11, 13 Mar 2010 Jan 2011 Jan 2012 May 2012
Growth April 2012 Page views per day · 12 Engineers · · · · · 1 Data Infrastructure 1 Ops 2 Mobile 8 Generalists 10 Non-Engineers Mar 2010 Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
Scaling Pinterest Scaling Pinterest Monday, November 11, 13
Growth April 2013 Page views per day April 2012 Scaling Pinterest Monday, November 11, 13 April 2013
Growth April 2013 Page views per day April 2012 Scaling Pinterest Monday, November 11, 13 April 2013
Growth April 2013 · · Page views per day Amazon EC2 + S3 + Edge Cast 400+ Web Engines + 400+ API Engines · 70+ MySQL DBs (hi.4xlarge on SSDs) + 1 slave each · · · 100+ Redis Instances 230+ Memcache Instances 10 Redis Task Manager + 500 Task Processors · 65+ Engineers (130+ total) April 2012 Scaling Pinterest Monday, November 11, 13 April 2013
Growth April 2013 · · Page views per day Amazon EC2 + S3 + Edge Cast 400+ Web Engines + 400+ API Engines · 70+ MySQL DBs (hi.4xlarge on SSDs) + 1 slave each · · · 100+ Redis Instances 230+ Memcache Instances 10 Redis Task Manager + 500 Task Processors · · · · · · · 65+ Engineers (130+ total) 8 services (80 instances) Sharded Solr 20 HBase 12 Kafka + Azkabhan 8 Zookeeper Instances 12 Varnish Scaling Pinterest Monday, November 11, 13 April 2012 April 2013
Growth April 2013 · 65+ Engineers · · · · · · · · · · Page views per day 7 Data Infrastructure + Science 7 Search and Discovery 9 Business and Platform 6 Spam, Abuse, Security 9 Web 9 Mobile 2 growth 10 Infrastructure 6 Ops 65+ Non-Engineers Scaling Pinterest Monday, November 11, 13 April 2012 April 2013
Scaling Pinterest Monday, November 11, 13
Technologies Scaling Pinterest Monday, November 11, 13
Arch Overview ELB Puppet StatsD Routing & Filtering (Varnish) Task Queue (Redis) Web App (Python) API App (Python / JS / HTML) Monit Sensu Task Processing (Python/Pyres) All connection pairings managed by ZooKeeper MySQL Service (Java/Finagle) Images (S3 + CDN) Scaling Pinterest Monday, November 11, 13 Memcache Mux (Nutcracker) Sharded MySQL Memcache Follower Service (Python/Thrift) Feed Service (Python/Thrift) Redis Search Service (Python/Thrift) HBase Spam Service (Python/Thrift)
Data Pipeline Web App (Python) API App (Python) Task Processing (Python/Pyres) Kafka S3 Copier Tripwire (Spam) S3 Qubole Pinball Scaling Pinterest Monday, November 11, 13 Redshift
Web App NGinX Website Rendering (x8) (Python / JS / HTML) API Scaling Pinterest Monday, November 11, 13
Our MySQL Sharding? http://www.infoq.com/presentations/Pinterest Scaling Pinterest Monday, November 11, 13
Choosing Your Tech Questions to ask • Does it meet your needs? • How mature is the product? • Is it commonly used? Can you hire people who have used it? • Is the community active? • How robust is it to failure? • How well does it scale? Will you be the biggest user? • Does it have a good debugging tools? Profiler? Backup software? • Is the cost justified? Scaling Pinterest Monday, November 11, 13
Hosting Why Amazon Web Services (AWS)? • Variety of servers running Linux • Very good peripherals: load balancing, DNS, map reduce, basic security, and more • Good reliability • Very active dev community • Not cheap, but... Scaling Pinterest Monday, November 11, 13
Hosting Why Amazon Web Services (AWS)? • Variety of servers running Linux • Very good peripherals: load balancing, DNS, map reduce, basic security, and more • Good reliability • Very active dev community • Not cheap, but... • New instances ready in seconds Scaling Pinterest Monday, November 11, 13
Hosting AWS Usage • Route 53 for DNS • ELB for 1st tier load balance • EC2 Ubuntu Linux • Varnish layer • All web, API, background appliances • All services • All databases and caches • S3 for images, logs Scaling Pinterest Monday, November 11, 13
Code Why Python? • Extremely mature • Well known and well liked • Solid active community • Very good libraries specifically targeted to web development • Effective rapid prototyping • Open Source Scaling Pinterest Monday, November 11, 13
Code Why Python? • Extremely mature • Well known and well liked • Solid active community • Very good libraries specifically targeted to web development • Effective rapid prototyping • Open Source Some Java and Go... • Faster, lower variance response time Scaling Pinterest Monday, November 11, 13
Code Python Usage • All web backend, API, and related business logic • Most services Scaling Pinterest Monday, November 11, 13
Code Python Usage • All web backend, API, and related business logic • Most services Java and Go Usage • Varnish plugins • Search indexers • High frequency services (e.g., MySQL service) Scaling Pinterest Monday, November 11, 13
Production Data Why MySQL and Memcache? • Extremely mature • Well known and well liked • (MySQL) Rarely catastrophic loss of data • Response time to request rate increases linearly • Very good software support: XtraBackup, Innotop, Maatkit • Solid active community • Open Source Scaling Pinterest Monday, November 11, 13
Production Data MySQL and Memcache Usage • Storage / Caching of core data • Users, boards, pins, comments, domains • Mappings (e.g., users to boards, user likes, repin info) • Legal compliance data Scaling Pinterest Monday, November 11, 13
Production Data Why Redis? • Well known and well liked • Active community • Consistently good performance • Variety of convenient and efficient data structures • 3 Flavors of Persistence: Now, Snapshot, Never • Open Source Scaling Pinterest Monday, November 11, 13
Production Data Redis Usage • Follower data • Configurations • Public feed pin IDs • Caching of various core mappings (e.g., board to pins) Scaling Pinterest Monday, November 11, 13
Production Data Why HBase? • Small, but growing loyal community • Difficult to hire for, but... • Non-volatile, O(1), extremely fast and efficient storage • Strong Hadoop integration • Consistently good performance • Used by Facebook (bigger than us) • Seems to work well • Open Source Scaling Pinterest Monday, November 11, 13
Production Data HBase Usage • User feeds (pin IDs are pushed to feeds) • Rich pin details • Spam features • User relationships to pins Scaling Pinterest Monday, November 11, 13
Production Data What happened to Cassandra, Mongo, ES, and Membase? • Does it meet your needs? • How mature is the product? • Is it commonly used? Can you hire people who have used it? • Is the community active? Can you get help? • How robust is it to failure? • How well does it scale? Will you be the biggest user? • Does it have a good debugging tools? Profiler? Backup software? • Is the cost justified? Scaling Pinterest Monday, November 11, 13
A 2nd chance... Scaling Pinterest Monday, November 11, 13
A 2nd Chance Stuff we could have done better • Logging on day 1 (StatsD, Kafka, Map Reduce) • Log every request, event, signup • Basic analytics • Recovery from data corruption or failure • Alerting on day 1 Scaling Pinterest Monday, November 11, 13
A 2nd Chance Stuff we could have done better • Shard our MySQL storage much earlier • Once you start relying on read slaves, start the timebomb countdown • We also fell into the NoSQL trap (Membase, Cassandra, Mongo, etc) • Pyres for background tasks day 1 • Hire technical operations eng earlier • Chef / Puppet earlier • Unit testing earlier (Jenkins for builds) Scaling Pinterest Monday, November 11, 13
A 2nd Chance Stuff we could have done better • A/B testing earlier • Decider on top of Zookeeper WATCH • Progressive roll out • Kill switches Scaling Pinterest Monday, November 11, 13
What’s next? Looking Forward • Continually improve Pinner experience • Help Pinners discover more of the things they love • Better uptime and lower latency • Faster development times • Reduce spam and abuse • Continually improve collaboration and build bigger, better, faster products • 180 Pinployees and beyond Scaling Pinterest Monday, November 11, 13
Have fun Scaling Pinterest Monday, November 11, 13
marty@pinterest.com pinterest.com/martaaay Monday, November 11, 13 yashh@pinterest.com pinterest.com/yashh
marty@pinterest.com pinterest.com/martaaay Monday, November 11, 13 yashh@pinterest.com pinterest.com/yashh
My 2nd Chance If I could do it all over again... • Stronger ACID transactional guarantees across multiple systems • Currently have: sometimes A, best effort C, I, D, no silent failure • Want: sometimes A, eventual C, I, D, no silent completion Scaling Pinterest Monday, November 11, 13
My 2nd Chance Transactional tasks • All tasks become a dependency tree of repeatable synchronous or asynchronous actions • All actions must be repeatable • Otherwise, must add repeatability • All tasks get a unique transaction number • Counters are tricky Scaling Pinterest Monday, November 11, 13
My 2nd Chance Transactional tasks • All tasks become a dependency tree of repeatable synchronous or asynchronous actions • Sync actions are executed in order • Async actions are executed in any order • Repeat until successful or too many failures • Too many failures -> put in per task failure queue • Gives eventual C, I, D • No silent completion and A require extra effort Scaling Pinterest Monday, November 11, 13
My 2nd Chance Transactional tasks example • Pin create sync • Write empty pin object • Write pin ID to board, likes, user’s pins, clear caches • Write pin object • Pin not shown until pin object created -> Atomicity! Scaling Pinterest Monday, November 11, 13
My 2nd Chance Transactional tasks example • Pin create async • Write pin to required user feeds and public feeds • Feeds are sorted sets. Reinsertion is okay. • Send emails, Facebook Likes, Twitter Tweets • Before send, check / record in temporary storage -> Gives (temporary) repeatability Scaling Pinterest Monday, November 11, 13
Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/scalingpinterest

More Related Content

PPT
Business Analyst Training
Craig Brown
 
PDF
SAAS Business Plan Example - CRM software
upmetrics.co
 
PDF
The Emerging Integration Reference Architecture | MuleSoft
MuleSoft
 
PPTX
Dev ops != Dev+Ops
Shalu Ahuja
 
DOCX
Micro services vs Monolith Architecture
MohamedElGohary71
 
PPTX
Practical Enterprise Architecture in Medium-size Corporation using TOGAF
Michael Sukachev
 
PPTX
Business Architecture Explained
aaronwilliamson
 
PDF
Full-Stack Development
Dhilipsiva DS
 
Business Analyst Training
Craig Brown
 
SAAS Business Plan Example - CRM software
upmetrics.co
 
The Emerging Integration Reference Architecture | MuleSoft
MuleSoft
 
Dev ops != Dev+Ops
Shalu Ahuja
 
Micro services vs Monolith Architecture
MohamedElGohary71
 
Practical Enterprise Architecture in Medium-size Corporation using TOGAF
Michael Sukachev
 
Business Architecture Explained
aaronwilliamson
 
Full-Stack Development
Dhilipsiva DS
 

What's hot (20)

PDF
Introduction to Business Process Model and Notation (BPMN) - OSSCamp 2014
OSSCube
 
PDF
Structured Approach to Solution Architecture
Alan McSweeney
 
PDF
Serverless and Design Patterns In GCP
Oliver Fierro
 
PPTX
Platform engineering 101
Sander Knape
 
PPTX
Open Digital Framework from TMFORUM
Maganathin Veeraragaloo
 
PDF
SaaS Introduction-May2014
Nguyen Tung
 
PDF
Introduction to ASP.NET Core
Avanade Nederland
 
PDF
Platform Engineering
Opsta
 
PDF
Microservices = Death of the Enterprise Service Bus (ESB)?
Kai Wähner
 
PPTX
DALLE-2.pptx
PIRSALMANSHAH
 
PPTX
Mendix Platform
SAKTHIVEL PERIYASAMY
 
PPTX
Introduction To Microservices
Lalit Kale
 
PDF
Necessity of the Digital Twin and Digital Thread
Marc Lind
 
PPT
Make Your API Catalog Essential with z/OS Connect EE
Teodoro Cipresso
 
PDF
Snyk Intro - Developer Security Essentials 2022
Liran Tal
 
PPT
Enterprise arhitecture blueprint objectives
Andy Parkins
 
PPTX
Full Stack Web Development
SWAGATHCHOWDARY1
 
PDF
Microservices
Trieu Nguyen
 
PDF
Role with IT(IL) - V3 Roles and Responsibilities - ITSM Academy Webinar
ITSM Academy, Inc.
 
PDF
Microservices Technology Stack
Eberhard Wolff
 
Introduction to Business Process Model and Notation (BPMN) - OSSCamp 2014
OSSCube
 
Structured Approach to Solution Architecture
Alan McSweeney
 
Serverless and Design Patterns In GCP
Oliver Fierro
 
Platform engineering 101
Sander Knape
 
Open Digital Framework from TMFORUM
Maganathin Veeraragaloo
 
SaaS Introduction-May2014
Nguyen Tung
 
Introduction to ASP.NET Core
Avanade Nederland
 
Platform Engineering
Opsta
 
Microservices = Death of the Enterprise Service Bus (ESB)?
Kai Wähner
 
DALLE-2.pptx
PIRSALMANSHAH
 
Mendix Platform
SAKTHIVEL PERIYASAMY
 
Introduction To Microservices
Lalit Kale
 
Necessity of the Digital Twin and Digital Thread
Marc Lind
 
Make Your API Catalog Essential with z/OS Connect EE
Teodoro Cipresso
 
Snyk Intro - Developer Security Essentials 2022
Liran Tal
 
Enterprise arhitecture blueprint objectives
Andy Parkins
 
Full Stack Web Development
SWAGATHCHOWDARY1
 
Microservices
Trieu Nguyen
 
Role with IT(IL) - V3 Roles and Responsibilities - ITSM Academy Webinar
ITSM Academy, Inc.
 
Microservices Technology Stack
Eberhard Wolff
 
Ad

Viewers also liked (17)

PDF
Red Hat Storage Server Replication Past, Present, & Future
Red_Hat_Storage
 
PDF
CloudFront DESIGN PATTERNS
Abhishek Tiwari
 
PPTX
Migrating enterprise workloads to AWS
Tom Laszewski
 
ODP
Bcache and Aerospike
Anshu Prateek
 
ODP
High Availability (HA) Explained - second edition
Maciej Lasyk
 
PDF
Cloud Native Cost Optimization
Adrian Cockcroft
 
PDF
From Push Technology to Real-Time Messaging and WebSockets
Alessandro Alinone
 
PDF
Datadog at NYCBUG
Alexis Lê-Quôc
 
ODP
Database index by Reema Gajjar
Reema Gajjar
 
PDF
Gripen: The Face of Success
Saab AB
 
PDF
Irwin Seating Lecture Room Brochure (2008 Edition)
LongoLabs
 
PDF
Portfolio
Mike Custer
 
PDF
Expertus 2010 Clipbook
Communications Strategy Group (CSG®)
 
PPTX
project management
Mohammad Noor
 
PDF
Masteel Accounting
Devin Wong
 
PDF
Luminous issue1 june_2008
David Aloi
 
PPTX
Informationsmöte Hagastaden 23 november
Hagastaden Stockholm
 
Red Hat Storage Server Replication Past, Present, & Future
Red_Hat_Storage
 
CloudFront DESIGN PATTERNS
Abhishek Tiwari
 
Migrating enterprise workloads to AWS
Tom Laszewski
 
Bcache and Aerospike
Anshu Prateek
 
High Availability (HA) Explained - second edition
Maciej Lasyk
 
Cloud Native Cost Optimization
Adrian Cockcroft
 
From Push Technology to Real-Time Messaging and WebSockets
Alessandro Alinone
 
Datadog at NYCBUG
Alexis Lê-Quôc
 
Database index by Reema Gajjar
Reema Gajjar
 
Gripen: The Face of Success
Saab AB
 
Irwin Seating Lecture Room Brochure (2008 Edition)
LongoLabs
 
Portfolio
Mike Custer
 
Expertus 2010 Clipbook
Communications Strategy Group (CSG®)
 
project management
Mohammad Noor
 
Masteel Accounting
Devin Wong
 
Luminous issue1 june_2008
David Aloi
 
Informationsmöte Hagastaden 23 november
Hagastaden Stockholm
 
Ad

Similar to Scaling Pinterest (20)

PDF
Plmce2012 scaling pinterest
Mohit Jain
 
PDF
Pinterest arch summit august 2012 - scaling pinterest
drewz lin
 
PDF
Lessons from Highly Scalable Architectures at Social Networking Sites
Patrick Senti
 
PDF
Scalable Architecture on Amazon AWS Cloud - Indicthreads cloud computing conf...
IndicThreads
 
PDF
Scaling web application in the Cloud
Federico Feroldi
 
PDF
The Reluctant SysAdmin : 360|iDev Austin 2010
Voxilate
 
PDF
Ivo jansch - the art of scalability
Tweakers.net Developer Summit
 
PDF
Running a business in the Cloud with AWS
Conor O'Neill
 
PDF
High Performance WordPress II
Barry Abrahamson
 
PPTX
Case study on Pinterest.pptx
Pratibha64549
 
DOCX
Architecting extremelylargescalewebapplications
Prashanth Panduranga
 
PDF
WordPress at Scale Webinar
Pantheon
 
PPTX
Flashback: QCon San Francisco 2012
Sergejus Barinovas
 
PPT
The Future is Now: Leveraging the Cloud with Ruby
Robert Dempsey
 
PPTX
CI_CONF 2012: Scaling - Chris Miller
The Huffington Post Tech Team
 
PPTX
CI_CONF 2012: Scaling
Chris Miller
 
PPTX
갑작스러운 유저의 수요 증가에 현명하게 대처하는 방법
Amazon Web Services Korea
 
PDF
OpenCms Days 2012 - The Dispatch - Running OpenCms 8 on Amazon cloud services
Alkacon Software GmbH & Co. KG
 
PPTX
Architecting extremelylarge scale web applications
Prashanth Panduranga
 
PDF
Rethinking the cloud_-_limitations_and_oppotunities_-_2011_nexcom
hybrid cloud
 
Plmce2012 scaling pinterest
Mohit Jain
 
Pinterest arch summit august 2012 - scaling pinterest
drewz lin
 
Lessons from Highly Scalable Architectures at Social Networking Sites
Patrick Senti
 
Scalable Architecture on Amazon AWS Cloud - Indicthreads cloud computing conf...
IndicThreads
 
Scaling web application in the Cloud
Federico Feroldi
 
The Reluctant SysAdmin : 360|iDev Austin 2010
Voxilate
 
Ivo jansch - the art of scalability
Tweakers.net Developer Summit
 
Running a business in the Cloud with AWS
Conor O'Neill
 
High Performance WordPress II
Barry Abrahamson
 
Case study on Pinterest.pptx
Pratibha64549
 
Architecting extremelylargescalewebapplications
Prashanth Panduranga
 
WordPress at Scale Webinar
Pantheon
 
Flashback: QCon San Francisco 2012
Sergejus Barinovas
 
The Future is Now: Leveraging the Cloud with Ruby
Robert Dempsey
 
CI_CONF 2012: Scaling - Chris Miller
The Huffington Post Tech Team
 
CI_CONF 2012: Scaling
Chris Miller
 
갑작스러운 유저의 수요 증가에 현명하게 대처하는 방법
Amazon Web Services Korea
 
OpenCms Days 2012 - The Dispatch - Running OpenCms 8 on Amazon cloud services
Alkacon Software GmbH & Co. KG
 
Architecting extremelylarge scale web applications
Prashanth Panduranga
 
Rethinking the cloud_-_limitations_and_oppotunities_-_2011_nexcom
hybrid cloud
 

More from C4Media (20)

PDF
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
C4Media
 
PDF
Next Generation Client APIs in Envoy Mobile
C4Media
 
PDF
Software Teams and Teamwork Trends Report Q1 2020
C4Media
 
PDF
Understand the Trade-offs Using Compilers for Java Applications
C4Media
 
PDF
Kafka Needs No Keeper
C4Media
 
PDF
High Performing Teams Act Like Owners
C4Media
 
PDF
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
C4Media
 
PDF
Service Meshes- The Ultimate Guide
C4Media
 
PDF
Shifting Left with Cloud Native CI/CD
C4Media
 
PDF
CI/CD for Machine Learning
C4Media
 
PDF
Fault Tolerance at Speed
C4Media
 
PDF
Architectures That Scale Deep - Regaining Control in Deep Systems
C4Media
 
PDF
ML in the Browser: Interactive Experiences with Tensorflow.js
C4Media
 
PDF
Build Your Own WebAssembly Compiler
C4Media
 
PDF
User & Device Identity for Microservices @ Netflix Scale
C4Media
 
PDF
Scaling Patterns for Netflix's Edge
C4Media
 
PDF
Make Your Electron App Feel at Home Everywhere
C4Media
 
PDF
The Talk You've Been Await-ing For
C4Media
 
PDF
Future of Data Engineering
C4Media
 
PDF
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
C4Media
 
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
C4Media
 
Next Generation Client APIs in Envoy Mobile
C4Media
 
Software Teams and Teamwork Trends Report Q1 2020
C4Media
 
Understand the Trade-offs Using Compilers for Java Applications
C4Media
 
Kafka Needs No Keeper
C4Media
 
High Performing Teams Act Like Owners
C4Media
 
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
C4Media
 
Service Meshes- The Ultimate Guide
C4Media
 
Shifting Left with Cloud Native CI/CD
C4Media
 
CI/CD for Machine Learning
C4Media
 
Fault Tolerance at Speed
C4Media
 
Architectures That Scale Deep - Regaining Control in Deep Systems
C4Media
 
ML in the Browser: Interactive Experiences with Tensorflow.js
C4Media
 
Build Your Own WebAssembly Compiler
C4Media
 
User & Device Identity for Microservices @ Netflix Scale
C4Media
 
Scaling Patterns for Netflix's Edge
C4Media
 
Make Your Electron App Feel at Home Everywhere
C4Media
 
The Talk You've Been Await-ing For
C4Media
 
Future of Data Engineering
C4Media
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
C4Media
 

Recently uploaded (20)

PDF
sbt 2.0: go big (Scala Days 2025 edition)
Eugene Yokota
 
PDF
Five Habits of High-Impact Board Members
OnBoard
 
PDF
How IoT Sensor Integration in 2025 is Transforming Industries Worldwide
Rejig Digital
 
PPTX
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
PDF
NewMind AI Weekly Chronicles – August ’25 Week III
NewMind AI
 
PDF
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
PDF
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
PPTX
Build Your First AI Agent with UiPath.pptx
suhanisingh58689
 
PDF
“A New Era of 3D Sensing: Transforming Industries and Creating Opportunities,...
Edge AI and Vision Alliance
 
PDF
Hybrid horned lizard optimization algorithm-aquila optimizer for DC motor
IAESIJAI
 
PDF
1 - Historical Antecedents, Social Consideration.pdf
tuazon2030627
 
PDF
Credit Without Borders: AI and Financial Inclusion in Bangladesh
Ehsan Tanim
 
PPTX
Configure Apache Mutual Authentication
Antonino Piraino
 
PPTX
Custom Battery Pack Design Considerations for Performance and Safety
Epec Engineered Technologies
 
PDF
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
PDF
Consumable AI The What, Why & How for Small Teams.pdf
Vivek Sai
 
PDF
OpenACC and Open Hackathons Monthly Highlights July 2025
OpenACC
 
PPTX
Benefits of Physical activity for teenagers.pptx
Nokwanda Thabethe
 
PDF
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
PPTX
TEXTILE technology diploma scope and career opportunities
kriti38
 
sbt 2.0: go big (Scala Days 2025 edition)
Eugene Yokota
 
Five Habits of High-Impact Board Members
OnBoard
 
How IoT Sensor Integration in 2025 is Transforming Industries Worldwide
Rejig Digital
 
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
NewMind AI Weekly Chronicles – August ’25 Week III
NewMind AI
 
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
Build Your First AI Agent with UiPath.pptx
suhanisingh58689
 
“A New Era of 3D Sensing: Transforming Industries and Creating Opportunities,...
Edge AI and Vision Alliance
 
Hybrid horned lizard optimization algorithm-aquila optimizer for DC motor
IAESIJAI
 
1 - Historical Antecedents, Social Consideration.pdf
tuazon2030627
 
Credit Without Borders: AI and Financial Inclusion in Bangladesh
Ehsan Tanim
 
Configure Apache Mutual Authentication
Antonino Piraino
 
Custom Battery Pack Design Considerations for Performance and Safety
Epec Engineered Technologies
 
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
Consumable AI The What, Why & How for Small Teams.pdf
Vivek Sai
 
OpenACC and Open Hackathons Monthly Highlights July 2025
OpenACC
 
Benefits of Physical activity for teenagers.pptx
Nokwanda Thabethe
 
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
TEXTILE technology diploma scope and career opportunities
kriti38
 

Scaling Pinterest

  • 1. Scaling Pinterest Marty Weiner Cloud Ninja Yash Nelapati Ascii Artist Monday, November 11, 13
  • 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /scaling-pinterest InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
  • 3. Presented at QCon San Francisco www.qconsf.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  • 5. Growth March 2010 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
  • 6. Growth March 2010 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
  • 7. Growth March 2010 Page views per day · · · · RackSpace 1 small Web Engine 1 small MySQL DB 1 Engineer + 2 Founders Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
  • 10. Growth January 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012
  • 11. Growth January 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012
  • 12. Growth January 2011 Page views per day · Amazon EC2 + S3 + CloudFront · · · 1 NGinX, 4 Web Engines 1 MySQL DB + 1 Read Slave 1 Task Queue + 2 Task Processors · · 1 MongoDB 2 Engineers + 2 Founders Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012
  • 14. Growth September 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
  • 15. Growth September 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
  • 16. Growth September 2011 Page views per day · · Amazon EC2 + S3 + CloudFront 2 NGinX, 16 Web Engines + 2 API Engines · 5 Functionally Sharded MySQL DB + 9 read slaves · · 4 Cassandra Nodes 15 Membase Nodes (3 separate clusters) · · · · · · 8 Memcache Nodes 10 Redis Nodes 3 Task Routers + 4 Task Processors 4 Elastic Search Nodes 3 Mongo Clusters 3 Engineers (8 Total) Scaling Pinterest Monday, November 11, 13 Mar 2010 Jan 2011 Jan 2012 May 2012
  • 17. It will fail. Keep it simple. Scaling Pinterest Monday, November 11, 13
  • 18. If you’re the biggest user of a technology, the challenges will be greatly amplified Scaling Pinterest Monday, November 11, 13
  • 20. Growth April 2012 Page views per day Mar 2010 Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
  • 21. Growth April 2012 Page views per day Mar 2010 Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
  • 22. Growth April 2012 Page views per day · · · · Amazon EC2 + S3 + Edge Cast 135 Web Engines + 75 API Engines 10 Service Instances 80 MySQL DBs (m1.xlarge) + 1 slave each · · · 110 Redis Instances 60 Memcache Instances 2 Redis Task Manager + 60 Task Mar 2010 Processors · 3rd party sharded Solr Scaling Pinterest Monday, November 11, 13 Mar 2010 Jan 2011 Jan 2012 May 2012
  • 23. Growth April 2012 Page views per day · 12 Engineers · · · · · 1 Data Infrastructure 1 Ops 2 Mobile 8 Generalists 10 Non-Engineers Mar 2010 Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
  • 25. Growth April 2013 Page views per day April 2012 Scaling Pinterest Monday, November 11, 13 April 2013
  • 26. Growth April 2013 Page views per day April 2012 Scaling Pinterest Monday, November 11, 13 April 2013
  • 27. Growth April 2013 · · Page views per day Amazon EC2 + S3 + Edge Cast 400+ Web Engines + 400+ API Engines · 70+ MySQL DBs (hi.4xlarge on SSDs) + 1 slave each · · · 100+ Redis Instances 230+ Memcache Instances 10 Redis Task Manager + 500 Task Processors · 65+ Engineers (130+ total) April 2012 Scaling Pinterest Monday, November 11, 13 April 2013
  • 28. Growth April 2013 · · Page views per day Amazon EC2 + S3 + Edge Cast 400+ Web Engines + 400+ API Engines · 70+ MySQL DBs (hi.4xlarge on SSDs) + 1 slave each · · · 100+ Redis Instances 230+ Memcache Instances 10 Redis Task Manager + 500 Task Processors · · · · · · · 65+ Engineers (130+ total) 8 services (80 instances) Sharded Solr 20 HBase 12 Kafka + Azkabhan 8 Zookeeper Instances 12 Varnish Scaling Pinterest Monday, November 11, 13 April 2012 April 2013
  • 29. Growth April 2013 · 65+ Engineers · · · · · · · · · · Page views per day 7 Data Infrastructure + Science 7 Search and Discovery 9 Business and Platform 6 Spam, Abuse, Security 9 Web 9 Mobile 2 growth 10 Infrastructure 6 Ops 65+ Non-Engineers Scaling Pinterest Monday, November 11, 13 April 2012 April 2013
  • 32. Arch Overview ELB Puppet StatsD Routing & Filtering (Varnish) Task Queue (Redis) Web App (Python) API App (Python / JS / HTML) Monit Sensu Task Processing (Python/Pyres) All connection pairings managed by ZooKeeper MySQL Service (Java/Finagle) Images (S3 + CDN) Scaling Pinterest Monday, November 11, 13 Memcache Mux (Nutcracker) Sharded MySQL Memcache Follower Service (Python/Thrift) Feed Service (Python/Thrift) Redis Search Service (Python/Thrift) HBase Spam Service (Python/Thrift)
  • 33. Data Pipeline Web App (Python) API App (Python) Task Processing (Python/Pyres) Kafka S3 Copier Tripwire (Spam) S3 Qubole Pinball Scaling Pinterest Monday, November 11, 13 Redshift
  • 34. Web App NGinX Website Rendering (x8) (Python / JS / HTML) API Scaling Pinterest Monday, November 11, 13
  • 36. Choosing Your Tech Questions to ask • Does it meet your needs? • How mature is the product? • Is it commonly used? Can you hire people who have used it? • Is the community active? • How robust is it to failure? • How well does it scale? Will you be the biggest user? • Does it have a good debugging tools? Profiler? Backup software? • Is the cost justified? Scaling Pinterest Monday, November 11, 13
  • 37. Hosting Why Amazon Web Services (AWS)? • Variety of servers running Linux • Very good peripherals: load balancing, DNS, map reduce, basic security, and more • Good reliability • Very active dev community • Not cheap, but... Scaling Pinterest Monday, November 11, 13
  • 38. Hosting Why Amazon Web Services (AWS)? • Variety of servers running Linux • Very good peripherals: load balancing, DNS, map reduce, basic security, and more • Good reliability • Very active dev community • Not cheap, but... • New instances ready in seconds Scaling Pinterest Monday, November 11, 13
  • 39. Hosting AWS Usage • Route 53 for DNS • ELB for 1st tier load balance • EC2 Ubuntu Linux • Varnish layer • All web, API, background appliances • All services • All databases and caches • S3 for images, logs Scaling Pinterest Monday, November 11, 13
  • 40. Code Why Python? • Extremely mature • Well known and well liked • Solid active community • Very good libraries specifically targeted to web development • Effective rapid prototyping • Open Source Scaling Pinterest Monday, November 11, 13
  • 41. Code Why Python? • Extremely mature • Well known and well liked • Solid active community • Very good libraries specifically targeted to web development • Effective rapid prototyping • Open Source Some Java and Go... • Faster, lower variance response time Scaling Pinterest Monday, November 11, 13
  • 42. Code Python Usage • All web backend, API, and related business logic • Most services Scaling Pinterest Monday, November 11, 13
  • 43. Code Python Usage • All web backend, API, and related business logic • Most services Java and Go Usage • Varnish plugins • Search indexers • High frequency services (e.g., MySQL service) Scaling Pinterest Monday, November 11, 13
  • 44. Production Data Why MySQL and Memcache? • Extremely mature • Well known and well liked • (MySQL) Rarely catastrophic loss of data • Response time to request rate increases linearly • Very good software support: XtraBackup, Innotop, Maatkit • Solid active community • Open Source Scaling Pinterest Monday, November 11, 13
  • 45. Production Data MySQL and Memcache Usage • Storage / Caching of core data • Users, boards, pins, comments, domains • Mappings (e.g., users to boards, user likes, repin info) • Legal compliance data Scaling Pinterest Monday, November 11, 13
  • 46. Production Data Why Redis? • Well known and well liked • Active community • Consistently good performance • Variety of convenient and efficient data structures • 3 Flavors of Persistence: Now, Snapshot, Never • Open Source Scaling Pinterest Monday, November 11, 13
  • 47. Production Data Redis Usage • Follower data • Configurations • Public feed pin IDs • Caching of various core mappings (e.g., board to pins) Scaling Pinterest Monday, November 11, 13
  • 48. Production Data Why HBase? • Small, but growing loyal community • Difficult to hire for, but... • Non-volatile, O(1), extremely fast and efficient storage • Strong Hadoop integration • Consistently good performance • Used by Facebook (bigger than us) • Seems to work well • Open Source Scaling Pinterest Monday, November 11, 13
  • 49. Production Data HBase Usage • User feeds (pin IDs are pushed to feeds) • Rich pin details • Spam features • User relationships to pins Scaling Pinterest Monday, November 11, 13
  • 50. Production Data What happened to Cassandra, Mongo, ES, and Membase? • Does it meet your needs? • How mature is the product? • Is it commonly used? Can you hire people who have used it? • Is the community active? Can you get help? • How robust is it to failure? • How well does it scale? Will you be the biggest user? • Does it have a good debugging tools? Profiler? Backup software? • Is the cost justified? Scaling Pinterest Monday, November 11, 13
  • 51. A 2nd chance... Scaling Pinterest Monday, November 11, 13
  • 52. A 2nd Chance Stuff we could have done better • Logging on day 1 (StatsD, Kafka, Map Reduce) • Log every request, event, signup • Basic analytics • Recovery from data corruption or failure • Alerting on day 1 Scaling Pinterest Monday, November 11, 13
  • 53. A 2nd Chance Stuff we could have done better • Shard our MySQL storage much earlier • Once you start relying on read slaves, start the timebomb countdown • We also fell into the NoSQL trap (Membase, Cassandra, Mongo, etc) • Pyres for background tasks day 1 • Hire technical operations eng earlier • Chef / Puppet earlier • Unit testing earlier (Jenkins for builds) Scaling Pinterest Monday, November 11, 13
  • 54. A 2nd Chance Stuff we could have done better • A/B testing earlier • Decider on top of Zookeeper WATCH • Progressive roll out • Kill switches Scaling Pinterest Monday, November 11, 13
  • 55. What’s next? Looking Forward • Continually improve Pinner experience • Help Pinners discover more of the things they love • Better uptime and lower latency • Faster development times • Reduce spam and abuse • Continually improve collaboration and build bigger, better, faster products • 180 Pinployees and beyond Scaling Pinterest Monday, November 11, 13
  • 57. marty@pinterest.com pinterest.com/martaaay Monday, November 11, 13 yashh@pinterest.com pinterest.com/yashh
  • 58. marty@pinterest.com pinterest.com/martaaay Monday, November 11, 13 yashh@pinterest.com pinterest.com/yashh
  • 59. My 2nd Chance If I could do it all over again... • Stronger ACID transactional guarantees across multiple systems • Currently have: sometimes A, best effort C, I, D, no silent failure • Want: sometimes A, eventual C, I, D, no silent completion Scaling Pinterest Monday, November 11, 13
  • 60. My 2nd Chance Transactional tasks • All tasks become a dependency tree of repeatable synchronous or asynchronous actions • All actions must be repeatable • Otherwise, must add repeatability • All tasks get a unique transaction number • Counters are tricky Scaling Pinterest Monday, November 11, 13
  • 61. My 2nd Chance Transactional tasks • All tasks become a dependency tree of repeatable synchronous or asynchronous actions • Sync actions are executed in order • Async actions are executed in any order • Repeat until successful or too many failures • Too many failures -> put in per task failure queue • Gives eventual C, I, D • No silent completion and A require extra effort Scaling Pinterest Monday, November 11, 13
  • 62. My 2nd Chance Transactional tasks example • Pin create sync • Write empty pin object • Write pin ID to board, likes, user’s pins, clear caches • Write pin object • Pin not shown until pin object created -> Atomicity! Scaling Pinterest Monday, November 11, 13
  • 63. My 2nd Chance Transactional tasks example • Pin create async • Write pin to required user feeds and public feeds • Feeds are sorted sets. Reinsertion is okay. • Send emails, Facebook Likes, Twitter Tweets • Before send, check / record in temporary storage -> Gives (temporary) repeatability Scaling Pinterest Monday, November 11, 13
  • 64. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/scalingpinterest