The journey of a symfony app from 150ms to 20ms
- 2. Alex Bumbacea ● Programmer since 2007 ● tackled a bit of sysadmin ● telecom/gaming/e-commerce ● Symfony user since 1.4
- 3. The journey of a Symfony app from 150ms to 20ms
- 5. More context ● Recommandations engine ● API with 2 endpoints ○ /bulk ○ /one
- 7. xhprof
- 8. Prep work ● Smoke tests - fast deployment ● Stable env ● Canary server (⅓ servers at least) ● Monitoring tools
- 9. Monitoring tools ● NewRelic ● Grafana ● ELK - monitor error rates
- 10. Daily status update Now Target 99% 4000ms 300ms 95% 3000ms N/A 50% - median 150ms N/A Throughput 100rps 800rps
- 12. Take 1 Gearman to parallelize bulk
- 13. Findings for take 1 ● Increase in throughput ● Bulk response time decreased ● Hard to scale fast ● Consuming a lot of memory
- 14. Take 2 Benchmark against java & node js (with gearman jobs in php)
- 15. Original stats
- 16. Take 2.1 Upgrade from php 5.6 to 7.0
- 17. Some more stats
- 19. Problems of take 2.1 ● Mongo driver support dropped ● Upgrade doctrine mongo bundle ● XHProf tools incompatibility
- 20. Findings take 2.1 25% decrease in request timings Memory usage reduction
- 21. Daily status update Before Now Target 99% 4000ms 3200ms 300ms 95% 3000ms 2000ms N/A 50% - median 150ms 100ms N/A Throughput 100rps 130rps 800rps
- 22. Take 3 Drop gearman and move to multi curl
- 23. Findings for take 3 ● Increase in throughput ● Better monitoring - bulk was hiding stuff ● Better http cache for our API -> we could use varnish in front of it
- 24. Daily status update Before Now Target 99% 3200ms 3000ms 300ms 95% 2000ms 2000ms N/A 50% - median 100ms 100ms N/A Throughput 130rps 200rps 800rps
- 25. Connecting / Reconnecting ● Redis ● Mongo ● MySQL ● etc ● Consider connection state (ex: redis selected db)
- 27. Consider your timeouts ● Solr / Mongo / MySQL / Redis etc ● None of your timeouts should be larger than your SLA
- 28. Network connection: Connection timeout!
- 31. Look into dispatched events ● It hides a lot of processing ● Not everything is required every time
- 32. Get rid of garbage - We removed bundles - SecurityBundle, SwiftMailerBundle, SensioFrameworkExtraBundle, TwigBundle, AsseticBundle - Only required services (that don’t have lazy loading) - Only required routes
- 33. Dedicated kernel (ApiKernel) - And dedicated cache folder :)
- 34. Autoload optimizations Composer optimize or not? For our project: NOT
- 35. IO
- 36. IO
- 37. IO
- 38. Queue it! If you don’t need it now… queue it :)
- 39. Slaves. Do you use them?
- 40. Config tweaks ● FPM children ++ ● Mongo max connections ++
- 41. Daily status Before Now Target 99% 4000ms 280ms 300ms 95% 3000ms 200ms N/A 50% - median 150ms 20ms N/A Throughput 100rps 500rps 800rps
- 42. Add servers X servers * 2
- 43. Daily status - END Before Target Now 99% 4000ms 300ms 280ms 95% 3000ms N/A 200ms 50% - median 150ms N/A 20ms Throughput 100rps 800rps 1100rps
- 46. Summary ● Gearman ● Java -> PHP 7 ● Gearman -> curl_multi_get ● Connections and timeouts ● ApiKernel ● IO / Memory / FPM ● Horizontal scaling
- 48. Don’t use those numbers on your app. Make your own tests.
- 50. Q&A