Database TCO
- 1. Total Cost of Ownership Zsolt Fabian Database Engineering
- 2. 2 1. Who is Spil Games? 2. What is TCO? 3. What drives the costs? 4. Possible improvements? Outline of the talk
- 3. Who are we? Who is Spil Games?
- 4. 4 • Company founded in 2001 • 350+ employees world wide • 200M+ unique visitors per month • 45 portals in 19 languages • Casual games • Social games • Real Kme mulKplayer games • Mobile games • 35+ MySQL clusters Facts
- 5. TCO Total Cost of Ownership
- 6. 6 • 1980 – “Life Cycle CosKng” • 1987 – Gartner Group • Total Cost of Ownership • Analysis of enterprise spending, vendor evaluaKon • Many inter-‐related direct and indirect costs • Problems with TCO What is TCO and why it is important? CAPEX – Capital Expenses OPEX – OperaKng Expenses Relevant Costs
- 8. 8 • CollecKng data • Different base systems • 4,3 TB of RAM • 402 drives wriKng 3 ExaBytes • 3 DBA • 12,3 kW power consumpKon • Asset value exceeds € 500,000 • Cables are available in all colors of the spectrum J A quick inventory
- 9. 9 Aging servers • Component failures • Power consumpKon • Volume of deployment
- 10. 10 • The most obvious • Base server + CPU + Mem + Disk + Controller • Replacement parts and Warranty • NPV – Net Present Value • Mining invoices CAPEX: Purchase costs AFR – Annual Failure Rate ALE – Annual Loss Expectancy
- 11. 11 • Business need • Cost of downKme • Levels of HA and Fault Tolerance • Server downKme • CPU failure • MySQLd crash • Solar acKvity • Combined availability formula: CAPEX: Purchase costs: HA A = 1 – (1 – Ax) ^ n
- 12. 12 • Giving back J • MySQL Standard EdiKon: $ 2,000 • Enterprise: $ 5,000 • Cluster Carrier Grade: $ 10,000 • Microsoj SQL Server: $ 6k+ / core • Oracle 10/11g: $ 40k+ • AddiKonal: Technology extensions, Dashboards CAPEX: License costs
- 13. 13 • Work hours spent… • SLA, Responsibility, Stress, Insider Threat • MySQL annual subscripKon: $ 2,000 • Specialist support from $ 1,500 / server / year • Unlimited servers for $ 30,000 / year • Depends on complexity • € 2,700 / server / year OPEX: MySQL Professional support It's cheaper to add more servers than hire more engineers.
- 14. 14 • Cage – Rack – Server • Networking • Deployment (+ fuel + Kme) • Power density constraints • Let’s say € 60 / server / month OPEX: Hosting: Colocation
- 15. 15 • Wao = Ampere * Volt • € 0.2 / kWh • 12,31 kWh (~ 16% of total) • € 21,300 annually • Measurements • IPMI • Power meter J • Contributors • HDD (25%) • Roles (5-‐10W) OPEX: Hosting: Power
- 16. 16 OPEX: Hosting: Power
- 18. Pu@ng it together The model Cost Item CAPEX OPEX TCO 5y Base System ? Extra memory ? Extra disks ? Licence fees ? Replacement parts / Warranty ? ? Co-‐locaKon ? Power ? Professional support ? TOTAL ? ? ?
- 19. Pu@ng it together Cluster of 6 nodes Cost Item (6x) CAPEX OPEX TCO 5y Purchase: HP DL380 G7 € 25,000 Purchase: Extra memory € 6,000 Professional support € 16,200 ColocaKon cost € 4,320 Power costs @200W € 2,100 Replacement parts € 1,500 TOTAL € 32,500 € 22,620 € 145,600
- 20. Pu@ng it together Cluster of 6 nodes Purchase costs 18% Professional support 58% ColocaKon 15% Power 8% Replacement parts 1% TCO 5 years
- 21. Improvements “Insanity: doing the same thing over and over again and expecSng different results” -‐ Albert Einstein
- 22. 22 Purchase prioritization Extend warranty MTBF Obsolete, Not Urgent Limited reparability Repairable New
- 23. 23 • SSD reduces the cost of IOPS • SSD is fast for reads, but not for writes • Has limited lifespan • NAND Flash and RAM based SSDs • MLC (MulK Level Cell) vs. SLC (Single Level Cell) • R/W in pages, but always erased in blocks • The main advantage that it doesn’t have to spin HDD vs. SSD: Inner workings
- 24. 24 • Dropping over Kme • Facts • Tsunami • Rare-‐earth minerals • Cost per GB is cheaper at HDD • Some predicts that in 5 to 10 years, SSD will replace HDD completely HDD vs. SSD: Pricing
- 25. 25 HDD vs. SSD: Power consumption Type Model Idle Power AcSve Power Annual cost contribuSon SSD Intel 520 600 mW 850 mW € 1,49 SSD Intel X25-‐M 60 mW 150 mW € 0,26 SSD Intel X25-‐E 60 mW 4,2 W € 7,35 HDD ST9146852S 5 W 7 W € 12,2
- 26. 26 • MTBF: ~ 200 years • HDD • AFR: 0,55% (-‐ 1,5% -‐ 2% -‐ …) • ALE! • SSD have LIMITED write cycles • SLC: 100,000 writes / block • MLC: 10,000 writes / block • Typical MLC: 5,000 writes / block • Future developments HDD vs. SSD: Endurance
- 27. 27 • Intel 520 SSD: 5 years (with normal desktop usage) • 35 TB write endurance (TBW) • Official endurance is overprovisioned (~ 50 %) • SaturaKon • Filesystem overhead (0.25 btrfs-‐> 0.015 (ext4)) • Readout • S.M.A.R.T.: Percentage of rated lifeKme used • UKliKes • RAID controllers HDD vs. SSD: Endurance
- 28. 28 • Determined during the design • The equaKon is about: Buffers x ConnecKons • DIMM populaKon guide • DRAM market: OVERSUPPLY Memory configuration
- 29. 29 • Extending Linux block-‐cache with SSD • Block-‐cache works by caching accessed blocks • Perfect sizing -‐> performance similar to SSD only • Failures • Writethrough cache || storage • Writearound –(w)–> storage –(r)–> cache –> • Writeback –(w)–> cache –(w)–> storage –> Flashcache
- 30. 30 • 6 x 300 GB RAID10 -‐> 2 x 100 GB SSD + 2 x 1 TB RAID1 • ConsideraKons • I/O rates • Write savings • Overprovisioning • ParKKoning • How long to cache • What to cache • Less HDD purchases, More SSD purchases (High ALE) Flashcache
- 31. 31 • 4 nodes of 6 x 146 GB HDD • CAPEX: € 4,700 • OPEX: € 293 • TCO 5y: € 6,165 • 4 nodes of 2 x 600 HDD + 2 x Intel 520 SSD, 30% opt. writes • Replacing them every year (TBWy: 100 TB) • CAPEX: € 11,680 • OPEX: € 110 • TCO: € 12,230 Flashcache: Case study
- 32. 32 • TCO is a very handy (someKmes misleading) tool • RCO -‐ Relevant Cost of Ownership • CAPEX:OPEX raKo • Giving back is OPEX • Keep eye on your power consumpKon • Buy DRAM • Consider ALE when you switch to SSD Summary
- 33. Questions?
- 34. 34 • PresentaKon can be found at: hop://spil.com/tcomeetup2013 • You can contact me at: zsolt.fabian@spilgames.com Thank you!