Startup DevOps - Jon Milsom Pitchero - Leeds DevOps - August 2014
0 likes843 views
The document discusses hiring needs for the startup Pitchero, which provides online management platforms for amateur sports teams. It highlights challenges faced with their hybrid cloud infrastructure, including latency and throughput issues, along with solutions for optimizing their newsletter system and caching strategies. The focus is on improving operational efficiency through better caching management and the adoption of DevOps practices in future developments.
Startup DevOps - Jon Milsom Pitchero - Leeds DevOps - August 2014
- 1. “Startup DevOps” Tuesday, 12th August 2014 Jon Milsom Co-‐founder & CTO
- 2. Hiring • Roles – FullHme Junior Dev (Frontend & some PHP) – Freelance SysAdmins / Ops for ad-‐hoc projects • Opportunity – Make a big impact in a small team
- 4. Websites for amateur sports teams
- 5. Online management plaRorm for amateur sports teams/clubs/ leagues/counHes/governing bodies
- 6. Market?
- 7. Not this…
- 9. This…
- 11. Traffic
- 13. iPhone Jan 2013
- 15. Tech Team
- 17. Hybrid Cloud? Image from: hWp://www.rackspace.com/cloud/hybrid/
- 18. Why Hybrid? • Reliable (keep some dedicated boxes) • Easy to scale
- 19. Moving from dedicated boxes… • Moved session management into Redis • Don’t rely on file system • Don’t rely on any single node • Rely on config files • Yada yada… Loads of other stuff that I can’t recall
- 21. Not cool L Source: hWp://www.mixcrate.com/oneworld/phenoms-‐of-‐funk-‐george-‐clintonparliamenRunkadelicbootsy-‐collins-‐162038
- 22. In pracHce Previously internal traffic now goes through firewall & back out
- 24. Issues -‐ Latency Approx 40ms memcache. ~ 20% of total!
- 26. Issues -‐ Throughput • Exceeded internal throughput – Increased latency – Dropped packets • Had to upgrade firewall – UnanHcipated cost
- 27. MESSAGE QUEUES
- 28. NewsleWer System • 1M emails / month • Up to 200k/day • Generated dynamically
- 29. NewsleWer System • Via Cron – Synchronous process – Sent via 5 minute cron – 500 in a batch
- 30. Problem?
- 31. 144,000 emails per day
- 35. SoluHon… High-‐Scale Async Processing The Message Queue For the Cloud
- 37. NewsleWer System • Via MQ – Processed Async – Sent quickly • 200k+ within 1 hour – X concurrent workers
- 38. NewsleWer System • Sessions • Average peak before: 5,000 • Average peak aver: 10,000 “Business” results…
- 39. CACHING
- 40. Simple Model
- 41. OpHmal Caching • Every client wants the same content • The content doesn’t change oven
- 42. OpHmal Caching • Every client wants the same content • The content doesn’t change oven
- 43. Pitchero Model
- 45. OpHon #1 – Increase DB layer • Pros – Simplify development • Cons – OperaHons headache – Expensive – Performance hit
- 46. OpHon #2 – Intelligent caching • Pros – Cheap (spare RAM?) – Fast – Low operaHons overhead • Cons – Dev work – Not relaHonal – Can’t rely on a key exisHng
- 47. “There are only two hard things in Computer Science: cache invalida:on and naming things.“ -‐ Phil Karlton
- 48. “There are only two hard things in Computer Science: cache invalida:on and naming things.“ -‐ Phil Karlton
- 49. ProperHes of our plaRorm • Natural Groups – PlaRorms • Club • League • County
- 50. ProperHes of our plaRorm • Natural Groups – PlaRorms • Club – Discreet club_ids: x,y,z • League – Discreet league_ids: i,j,k • County – Discreet county_ids: p,q,r
- 51. SoluHon #1 “The InvalidaHon Manager” • ApplicaHon – “I have changed A” • InvalidaHon Manager – “I will invalidate A1, A2, A3”
- 53. What really happened… • ApplicaHon – “I have changed A” • InvalidaHon Manager – “I will invalidate A1, A2, A3, B2, C7, D4, F8, G8, A3 (twice!), B6, H28, M14, J139”
- 54. “The InvalidaHon Manager” Un-‐managable!
- 55. Problems • Cross domain dependencies A, B, C etc • Reliance on developers • Large number of invalidaHon events • Users content disappeared • Users could not delete content
- 56. SoluHon #2 – “PlaRorm Cache” • Example cache keys – “club1234_news56789” – “club1234_report4321” – “club1234_video1357”
- 57. SoluHon #2 – “PlaRorm Cache” • Example cache keys – “club1234_news56789” – “club1234_report4321” – “club1234_video1357”
- 58. SoluHon #2 – “PlaRorm Cache” • Example cache keys – “{club}_news56789” – “{club}_report4321” – “{club}_video1357” • {club} is a variable, stored in memcache • {club} = hash_fn(club, club_id, Hme())
- 59. Just added support for dynamically namespacing memcache keys… Source: hWp://www.troll.me/meme/fist-‐baby
- 60. SoluHon #2 – “PlaRorm Cache” • EffecHvely namespaced cache keys • Can now invalidate* an enHre plaRorms (e.g. club_id:1234) data in one command • No required knowledge of the rest of the key • Don’t need to add keys to invalidaHon manager each Hme we add new funcHonality *No actual invalidaHon happens: we just change the names of the keys to something that doesn’t exist. Memcache will garbage collect based on LRU.
- 61. SoluHon #2 – “PlaRorm Cache” • Results – Reduced load on database – Low ops overhead – LiWle input required from developers
- 62. SoluHon #2 – “PlaRorm Cache” • Not perfect – Requires an extra memcache lookup for the namespace key
- 67. Any image
- 68. Any size
- 69. As the page loads
- 70. Problems • 10M images • Currently in 3 sizes (30M objects) • Up to 50 images on some pages – High traffic => very high request rate
- 71. Old Model 3 set sizes /{user_id}/{image_id}.jpg -‐ 600x450 /{user_id}/sm_{image_id}.jpg -‐ 120x90 /{user_id}/Hny_{image_id}.jpg -‐ 50x37
- 72. New Model Any image, any size /?url={s3_image_url}&w={width}&h={height}&q={quality} Hash parameters and store on disk once generated Checkout: hWp://images.weserv.nl/
- 73. New Model Super fast… Average server response 15ms (CloudFront faster?) >> CloudFront >> Server (from disk) >> Server (generate from S3 original)
- 75. DevOps Status • Dev working pracHces – Version control – One command deployment – (Some) automated tesHng • Ops working pracHces in infancy – Setup lists & bash scripts – Not taking advantage of the cloud
- 76. The Future • Infrastructure as code • More automated tesHng • BeWer automated tesHng • Cloud? Dedicated?
- 77. QuesHons? @jonmilsom jon@pitchero.com hWp://uk.linkedin.com/in/jonmilsom