Qcon 090408233824-phpapp01
- 2. Facebook Architecture Aditya Agarwal Director of Engineering 11/22/2008
- 3. Agenda 1 Architecture Overview 2 PHP, MySQL, Memcache 3 Thrift, Scribe, Tools 4 News Feed Architecture
- 4. At a Glance The Social Graph 120M+ active users 50B+ PVs per month 10B+ Photos 1B+ connections 50K+ Platform Apps 400K+ App Developers
- 5. General Design Principles ▪ Use open source where possible ▪ Explore making optimizations where needed ▪ Unix Philosophy ▪ Keep individual components simple yet performant ▪ Combine as necessary ▪ Concentrate on clean interface points ▪ Build everything for scale ▪ Try to minimize failure points ▪ Simplicity, Simplicity, Simplicity!
- 6. Architecture Overview LAMP + Services PHP AdServer Search Memcache Network Selector News Feed MySQL Blogfeeds CSSParser php! Mobile ShareScraper !php Thrift Scribe ODS Tools
- 7. PHP ▪ Good web programming language ▪ Extensive library support for web development ▪ Active developer community ▪ Good for rapid iteration ▪ Dynamically typed, interpreted scripting language
- 8. PHP: What we Learnt ▪ Tough to scale for large code bases ▪ Weak typing ▪ Limited opportunities for static analysis, code optimizations ▪ Not necessarily optimized for large website use case ▪ E.g. No dynamic reloading of files on web server ▪ Linearly increasing cost per included file ▪ Extension framework is difficult to use
- 9. PHP: Customizations ▪ Op-code optimization ▪ APC improvements ▪ Lazy loading ▪ Cache priming ▪ More efficient locking semantics for variable cache data ▪ Custom extensions ▪ Memcache client extension ▪ Serialization format ▪ Logging, Stats collection, Monitoring ▪ Asynchronous event-handling mechanism
- 10. MySQL ▪ Fast, reliable ▪ Used primarily as <key,value> store ▪ Data randomly distributed amongst large set of logical instances ▪ Most data access based on global id ▪ Large number of logical instances spread out across physical nodes ▪ Load balancing at physical node level ▪ No read replication
- 11. MySQL: What We Learnt (ing) ▪ Logical migration of data is very difficult ▪ Create a large number of logical dbs, load balance them over varying number of physical nodes ▪ No joins in production ▪ Logically difficult (because data is distributed randomly) ▪ Easier to scale CPU on web tier
- 12. MySQL: What we Learnt (ing) ▪ Most data access is for recent data ▪ Optimize table layout for recency ▪ Archive older data ▪ Don’t ever store non-static data in a central db ▪ CDB makes it easier to perform certain aggregated queries ▪ Will not scale ▪ Use services or memcache for global queries ▪ E.g.: What are the most popular groups in my network
- 13. MySQL: Customizations ▪ No extensive native MySQL modifications ▪ Custom partitioning scheme ▪ Global id assigned to all data ▪ Custom archiving scheme ▪ Based on frequency and recency of data on a per-user basis ▪ Extended Query Engine for cross-data center replication, cache consistency
- 14. MySQL: Customizations ▪ Graph based data-access libraries ▪ Loosely typed objects (nodes) with limited datatypes (int, varchar, text) ▪ Replicated connections (edges) ▪ Analogous to distributed foreign keys ▪ Some data collocated ▪ Example: User profile data and all of user’s connections ▪ Most data distributed randomly
- 15. Memcache ▪ High-Performance, distributed in-memory hash table ▪ Used to alleviate database load ▪ Primary form of caching ▪ Over 25TB of in-memory cache ▪ Average latency < 200 micro-seconds ▪ Cache serialized PHP data structures ▪ Lots and lots of multi-gets to retrieve data spanning across graph edges
- 16. Memache: Customizations ▪ Memache over UDP ▪ Reduce memory overhead of thousands of TCP connection buffers ▪ Application-level flow control (optimization for multi-gets) ▪ On demand aggregation of per-thread stats ▪ Reduces global lock contention ▪ Multiple Kernel changes to optimize for Memcache usage ▪ Distributing network interrupt handling over multiple cores ▪ Opportunistic polling of network interface
- 17. Let’s put this into action
- 18. Under the Covers ▪ Get my profile data ▪ Fetch from cache, potentially go to my DB (based on user-id) ▪ Get friend connections ▪ Cache, if not DB (based on user-id) ▪ In parallel, fetch last 10 photo album ids for each of my friends ▪ Multi-get; individual cache misses fetches data from db (based on photo- album id) ▪ Fetch data for most recent photo albums in parallel ▪ Execute page-specific rendering logic in PHP ▪ Return data, make user happy
- 19. LAMP is not Perfect
- 20. LAMP is not Perfect ▪ PHP+MySQL+Memcache works for a large class of problems but not for everything ▪ PHP is stateless ▪ PHP not the fastest executing language ▪ All data is remote ▪ Reasons why services are written ▪ Store code closer to data ▪ Compiled environment is more efficient ▪ Certain functionality only present in other languages
- 21. Services Philosophy ▪ Create a service iff required ▪ Real overhead for deployment, maintenance, separate code-base ▪ Another failure point ▪ Create a common framework and toolset that will allow for easier creation of services ▪ Thrift ▪ Scribe ▪ ODS, Alerting service, Monitoring service ▪ Use the right language, library and tool for the task
- 22. Thrift High-Level Goal: Enable transparent interaction between these. …and some others too.
- 23. Thrift ▪ Lightweight software framework for cross-language development ▪ Provide IDL, statically generate code ▪ Supported bindings: C++, PHP, Python, Java, Ruby, Erlang, Perl, Haskell etc. ▪ Transports: Simple Interface to I/O ▪ Tsocket, TFileTransport, TMemoryBuffer ▪ Protocols: Serialization Format ▪ TBinaryProtocol, TJSONProtocol ▪ Servers ▪ Non-Blocking, Async, Single Threaded, Multi-threaded
- 24. Hasn’t this been done before? (yes.) ▪ SOAP ▪ XML, XML, and more XML ▪ CORBA ▪ Bloated? Remote bindings? ▪ COM ▪ Face-Win32ClientSoftware.dll-Book ▪ Pillar ▪ Slick! But no versioning/abstraction. ▪ Protocol Buffers
- 25. Thrift: Why? • It’s quick. Really quick. • Less time wasted by individual developers • No duplicated networking and protocol code • Less time dealing with boilerplate stuff • Write your client and server in about 5 minutes • Division of labor • Work on high-performance servers separate from applications • Common toolkit • Fosters code reuse and shared tools
- 26. Scribe ▪ Scalable distributed logging framework ▪ Useful for logging a wide array of data ▪ Search Redologs ▪ Powers news feed publishing ▪ A/B testing data ▪ Weak Reliability ▪ More reliable than traditional logging but not suitable for database transactions. ▪ Simple data model ▪ Built on top of Thrift
- 27. Other Tools ▪ SMC (Service Management Console) ▪ Centralized configuration ▪ Used to determine logical service -> physical node mapping
- 28. Other Tools ▪ ODS ▪ Used to log and view historical trends for any stats published by service ▪ Useful for service monitoring, alerting
- 29. Open Source ▪ Thrift ▪ http://developers.facebook.com/thrift/ ▪ Scribe ▪ http://developers.facebook.com/scribe/ ▪ PHPEmbed ▪ http://developers.facebook.com/phpembed/ ▪ More good stuff ▪ http://developers.facebook.com/opensource.php
- 30. NewsFeed – The Goodz
- 31. NewsFeed – The Work friends’ actions web tier Leaf Server Html PHP Actions (Scribe) Leaf Server home.php Leaf Server user return Leaf Server view state view aggregators state storage friends’ actions? aggregating... - Most arrows indicate thrift calls ranking...
- 32. Search – The Goodz
- 33. Search – The Work Thrift search tier slave slave master slave index index index index user web tier Scribe live db PHP change index logs files Indexing service DB Tier Updates
- 34. Questions? More info at www.facebook.com/eblog Aditya Agarwal aditya@facebook.com