What Happens When You Type en.wikipedia.org - SREcon19 EMEA
1 like473 views
The document provides an overview of Wikimedia Foundation's infrastructure, emphasizing its open-source software, server architecture, and operational statistics. It highlights the transition from a monolithic application to microservices and discusses the technologies used, including MediaWiki, MariaDB, Apache Kafka, and Kubernetes. The document also addresses the foundation's volunteer-driven nature and its global reach in supporting a multitude of languages.
What Happens When You Type en.wikipedia.org - SREcon19 EMEA
- 1. what happens when you type en.wikipedia.org SREcon19 Dublin @kosiaris • @manjiki effie mouzeli • alexandros kosiaris
- 2. About CC BY-SA 4.0 Niccolò Caranti@kosiaris • @manjiki 2@kosiaris • @manjiki
- 3. Did you know... ● … the Wikipedia infrastructure is run by the Wikimedia Foundation, an American nonprofit charitable organisation? ● … and we are ~370 people? ● … and we have no affiliation with Wikileaks? ● … all content is managed by volunteers? ● … we support 304 languages? ● … Wikipedia is 18 years old ? ● … Wikipedia hosts some really really weird articles? ● … which can’t be read in Turkey (2017) nor China (2019)? 3
- 5. Wikimedia Infrastructure ✺ Open source software ✺ 2 Primary Data Centres ✺ 3 Caching Points of Presence ✺ ~17 billion pageviews per month* ✺ ~300k new editors per month ✺ ~1300 bare metal servers 5* it’s complicated
- 6. Site Reliability Engineering ✺ Datacenter Operations ✺ Data Persistence ✺ Infrastructure Foundations ✺ Service Operations ✺ Traffic The SRE team is a globally distributed team of 26 people responsible for developing and maintaining Wikimedia's production systems The Foundation has more SREs in other teams as well! 6
- 7. Application Layer CC BY-SA 2.0 Arthur Dunn@kosiaris • @manjiki 7
- 8. MediaWiki ✺ Our core application ✺ PHP, Apache, MySQL. Yes.* ✴ PHP7.2 since Sept 2019 ✺ Wiki web pages - app servers cluster ✺ API cluster ✺ Jobrunners/Videoscalers cluster MediaWiki is a free server-based software, licensed under the GNU GPL. It is an extremely powerful, scalable software, and a feature-rich wiki implementation that uses PHP to process and display data stored in a database, such as MySQL. 8* it’s complicated
- 11. ✺ Elasticity ✺ Hardware fault mitigation ✺ Deployments ✺ Migration is not easy, and still ongoing 11 From a Monolith to Microservices
- 12. Microservices! ✺ Thumbor ✺ Mathoid ✺ ORES ✺ Mobile Content Service (MCS) ✺ And many more Thumbor is used for imagescaling Mathoid renders LaTeX, and returns JSON with PNG, SVG or MathML renderings of the formula ORES scores edits using Machine Learning (anti-vandalism effort) MCS modifies page content on the fly, tailoring it for mobile 12
- 13. Kubernetes Public Domain@kosiaris • @manjiki
- 14. ✺ Bare metal ✺ Calico as a CNI plugin ✺ Helm for deployments ✺ 2 clusters + 1 staging one ✺ Docker as a CRE We have been running it successfully for the last 2 years! Currently, 11 services on it. Got a pipeline in the works. Powers all mathematical formulas on Wikipedia!!! 14 Kubernetes
- 15. Message Queueing CC BY 2.0 bootbearwdc@kosiaris • @manjiki
- 16. Message Queueing ✺ Yes, we use Apache Kafka ✺ We are sending events like: ✳ wikitext templates refresh ✳ edge caches purging ✳ cross wiki links ✳ create new thumbnails ✳ re-encoding videos to open source formats Apache Kafka: stream processing platform for real-time data feeds One message queue to rule them all; started as a service for Analytics only. Now, it is our de facto solution. 16
- 17. Databases CC BY 2.0 RageZ@kosiaris • @manjiki 17
- 18. MariaDB* ✺ Database clusters are divided into sections ✺ Sections have masters and replicas* ✺ MediaWiki reads from replicas and writes to master ✺ Clusters: ✳ Wikitext (compressed) ✳ Metadata ✳ Parsercache MariaDB: fork of MySQL, migrated from MySQL in 2013* Have a go at https://quarry.wmflabs.org 18* it’s complicated
- 19. MariaDB 19 ✺ Online schema migrations* ✺ Cross DC replication ✺ TLS across all DBs ✺ Snapshots and local dumps for Backups ✺ ~570 TB total data ✺ ~150 DB servers ✺ ~350k queries per second (qps) ✺ ~70 TB of RAM * it’s complicated
- 20. Elasticsearch You guessed it right, we use it for search. That box on your top right. Run by a team surprisingly called Search Platform! 20
- 21. Storage CC BY-NC 2.0 Gail Thomas@kosiaris • @manjiki
- 22. Swift ✺ All our media are stored on Swift ✺ It has frontends … and backends ✺ 1 billion objects ✺ ~390 TB of media! OpenStack Object Storage: a scalable storage system that stores and retrieves data via HTTP 22
- 23. Traffic Public Domain@kosiaris • @manjiki 23
- 24. Network 24
- 25. Network 25
- 26. ✺ We have our own content delivery network ✺ We direct traffic to a location on demand (via GeoDNS) ✳ Pooling/Depooling DCs ✳ 10 min TTL ✺ LVS as a Layer 3/4 Linux loadbalancer* gdnsd: GeoDNS is written and maintained by one of us peering: interconnection with other internet networks Linux Virtual Server: an advanced L3/L4 load balancing solution for linux, supports consistent hashing pybal: LVS manager, developed in-house Network 26* it’s complicated
- 27. LVS-DR 27
- 28. CDN 28
- 29. Nginx-: Highly performant HTTP webserver/proxy with excellent TLS support Varnish: Reverse HTTP caching proxy CDN 29 ✺ Nginx- for TLS termination ✺ Varnish frontend ✳ in memory ✺ Varnish backend ✳ local stores ✺ Varnish text ✳ HTML, CSS, JS etc ✺ Varnish upload ✳ media, media, media
- 31. Apache Traffic Server: Reverse and forward proxy with excellent caching support ACME-chief: handles all the process of issuing and renewing Let’s Encrypt certificates (dns-01) CDN (coming soon) 31 ✺ ATS TLS ✳ in memory ✺ ATS backend ✳ local store (SSDs) ✺ ATS text ✳ HTML, CSS, JS etc ✺ ATS upload ✳ media, media, media ✺ ACME-chief
- 32. what happens when you type en.wikipedia.org @manjiki • @kosiaris CC BY 3.0 WikiReader
- 33. Read (cached) 33
- 34. Read (cached) 34
- 36. Edit - Media Upload 36
- 37. Managing to Manage GETTY IMAGES@kosiaris • @manjiki
- 38. ✺ Infrastructure as code ✺ Configuration management ✺ Kubernetes ✺ Testing/CI/CD ✺ Orchestration tooling Puppet: configuration management system for servers/services ...~50k lines of puppet code ...~100k lines of Ruby/ERB Cumin: in-house automation and orchestration tool Managing to Manage 38
- 39. @kosiaris • @manjiki In a Nutshell CC BY 2.0 Peter Trimming
- 40. Want to sell encyclopedias? https://jobs.wikimedia.org https://grafana.wikimedia.org/ https://github.com/wikimedia/operations-puppet https://phabricator.wikimedia.org/ https://wikitech.wikimedia.org/ SREcon19 Dublin @kosiaris • @manjiki