Taras Matsyk - Event Driven ML
0 likes49 views
This document summarizes the development of an image processing API and ML system to meet requirements of processing images in under 2 seconds, with visibility, persistence, scalability, and extensibility. It describes splitting the API and ML into separate services, optimizing processing times from 4 to 1 second through changes like using Kafka, scaling the system, and troubleshooting issues with performance spikes, message brokers, and databases. Lessons learned include distinguishing CPU from IO-bound tasks and the challenges of Go versus Python. The overall goal of under 1 second per request processing time was eventually achieved.
Taras Matsyk - Event Driven ML
- 1. Go && Python Event driven ML Lalafo case
- 3. How it all started Part 1. The Phantom Menace
- 5. StackOverflow: Ready to rescue
- 6. 200 RPM
- 8. $ Feedback
- 9. $ Feedback Processes Monitoring RAM - expiration? Relational DB?
- 10. Any issues there? 1. Request monitoring (?) 2. Hard to reason GPU usage (varies 3 to 7GB) 3. ~5Gb RAM out of the box (scale?) 4. Redis is in-memory storage (temporary) 5. Threading != Parallel
- 12. What if we split API and ML Part 2. A new hope
- 13. Requirements Lalafo case 1. Single image/request processing time: < 2 seconds 2. Visibility 3. Persistence 4. Scalability 5. Make it extendable for new features a. Price prediction b. Similarity search c. Segmentation 6. SDK friendly (well documented, tested etc)
- 14. $ Feedback Processes Monitoring RAM - expiration? Relational DB?
- 15. After 2 months API Listeners Feedback $
- 16. Requirements Lalafo case 1. Single image/request processing time: < 2 seconds 2. Visibility - Decouple request and prediction 3. Persistence 4. Scalability 5. Make it extendable for new features a. Price prediction b. Similarity search c. Segmentation 6. SDK friendly (well documented, tested etc)
- 17. 2 seconds per request Part 3. The Empire Strikes Back
- 18. 4 seconds per request API Listeners 1 second 0.01 sec
- 19. segmentio/kafka-go -> shopify/sarama API Listeners 0.02 sec 0.01 sec
- 20. 3 seconds per request API Listeners 0.02 sec 0.3 second 0.3 second Synchronous request
- 21. How large is every image
- 22. 2.4 seconds per request API Listeners 0.02 sec Synchronous request
- 23. 2.4 seconds per request API Listeners 0.02 sec Synchronous request
- 24. 2 seconds per request API Listeners 0.02 sec Synchronous request Add ThreadPool
- 25. 2 seconds per request Success? Not yet
- 26. 3 seconds per request API Listeners 0.02 sec
- 27. 3 seconds per request API Listeners 0.02 sec 1 second and growing
- 28. 3 seconds per request
- 29. 2 seconds per request
- 30. ¯_(ツ)_/¯
- 31. 2 seconds per request API Listeners 0.02 sec
- 32. 200 RPM -> 1200 RPM
- 33. 2 seconds per request API Listeners 0.02 sec Scale Scale Master -> Slave -> balancer Topics -> Partitions
- 34. < 1 second per request Success? Not yet
- 36. Out of 8Gb
- 37. How to PyTorch in production
- 38. 7.6 Gb -> 600 Mb
- 39. 1 second per request Success? Not yet
- 40. Back to 2, 5, 10 seconds per request
- 43. Troubleshooting API Listeners 0.02 sec ? 175k records per request
- 46. SELECT * FROM ads LIMIT 1
- 47. 25 seconds -> 0.02 seconds
- 48. 0.02 seconds -> 0.002 second
- 49. Success? Not yet
- 50. Faust 1.4.6: No latest offset https://github.com/robinhood/faust
- 51. 3K images per minute 0_o
- 52. Success? Not yet
- 53. Issues to solve 1. Occasional spikes in performance (GC, network latency) 2. Message broker (Kafka rebalancing, offset etc) 3. How to handle DB migrations 4. Something we are not aware of yet
- 54. Lessons learnt 1. CPU bound tasks != IO bound (¯_(ツ)_/¯) 2. High coupling - low cohesion 3. You need to know how to cook MongoDB 4. Go is not that obvious and library reach as Python 5. Simple != Easier 6. Concurrency != Parallelism (obviously)
- 55. < 1 second per request
- 56. + Live statistics from PostgreSQL API Listeners