Scalling Rails: The Journey to 200M Notifications
0 likes317 views
Talk presented at Tropical On Rails.
Scalling Rails: The Journey to 200M Notifications
- 1. SCALING RAILS The Journey to 200M Notifications
- 2. Software Engineer @ CloudWalk gustavoaraujo.dev garaujodev garaujodev Gustavo Araujo
- 3. Software Engineer @ CloudWalk 🎯 Focused on Ruby on Rails and Elixir gustavoaraujo.dev garaujodev garaujodev Gustavo Araujo
- 4. Software Engineer @ CloudWalk 🎯 Focused on Ruby on Rails and Elixir 💜 Passionate about code quality, observability, and performance gustavoaraujo.dev garaujodev garaujodev Gustavo Araujo
- 5. Software Engineer @ CloudWalk 🎯 Focused on Ruby on Rails and Elixir 💜 Passionate about code quality, observability, and performance Sharing knowledge through talks and technical content gustavoaraujo.dev garaujodev garaujodev Gustavo Araujo
- 6. Application Challenges ➡ High-Volume Notifications (1B last year)
- 7. Application Challenges ➡ High-Volume Notifications (1B last year) ➡ Multi-channel (e.g., WhatsApp, SMS, Email, Push)
- 8. Application Challenges ➡ High-Volume Notifications (1B last year) ➡ Several Database Writes ➡ Multi-channel (e.g., WhatsApp, SMS, Email, Push)
- 9. Application Challenges ➡ High-Volume Notifications (1B last year) ➡ Several Database Writes ➡ Multi-channel (e.g., WhatsApp, SMS, Email, Push) ➡ Unpredictable workloads
- 10. The challenges we faced, and the key lessons learned
- 11. CONCEPT OF SCALE
- 12. CONCEPT OF SCALE
- 13. HOW WE STARTED?
- 15. ⚙Concurrency Model ⚡ Performance ✅ Best Use Case ❌ Limitations WebRick Single-threaded + single-process Very low Development only Not production-ready; removed in Rails 6+ Ruby Webservers
- 16. ⚙Concurrency Model ⚡ Performance ✅ Best Use Case ❌ Limitations WebRick Single-threaded + single-process Very low Development only Not production-ready; removed in Rails 6+ Passenger Multi-threaded + multi-process (hybrid) Medium to High Easy-to-deploy production apps Advanced features require paid license Ruby Webservers
- 17. ⚙Concurrency Model ⚡ Performance ✅ Best Use Case ❌ Limitations WebRick Single-threaded + single-process Very low Development only Not production-ready; removed in Rails 6+ Passenger Multi-threaded + multi-process (hybrid) Medium to High Easy-to-deploy production apps Advanced features require paid license Unicorn Multi-process (no threads) High for CPU-bound apps Apps requiring process-level isolation No concurrency per worker; not ideal for I/O-bound apps Ruby Webservers
- 18. ⚙Concurrency Model ⚡ Performance ✅ Best Use Case ❌ Limitations WebRick Single-threaded + single-process Very low Development only Not production-ready; removed in Rails 6+ Passenger Multi-threaded + multi-process (hybrid) Medium to High Easy-to-deploy production apps Advanced features require paid license Unicorn Multi-process (no threads) High for CPU-bound apps Apps requiring process-level isolation No concurrency per worker; not ideal for I/O-bound apps Puma Multi-threaded + multi-process (clustered) Highly configurable — scales with threads and workers Modern Rails apps needing concurrent I/O handling Requires tuning (threads/workers + DB pool alignment) Ruby Webservers
- 19. Puma ✅ ❌ Clustered Mode Best for Multi-core CPUs, CPU-bound applications Higher memory usage since each worker is a separate process
- 20. Puma ✅ ❌ Clustered Mode Best for Multi-core CPUs, CPU-bound applications Higher memory usage since each worker is a separate process Single Mode Best for I/O-bound apps, lower memory usage Limitation: Single process, may not fully utilize multi-core CPUs
- 21. How Many Workers? Scenario Recommendation 🖥 App runs on VM/bare metal Use 1 worker per available CPU core
- 22. How Many Workers? Scenario Recommendation 🖥 App runs on VM/bare metal Use 1 worker per available CPU core 📦 App runs in Docker/Kubernetes Respect container CPU limit (cpu_limit = 2) → set workers = 2
- 23. How Many Workers? Scenario Recommendation 🖥 App runs on VM/bare metal Use 1 worker per available CPU core 📦 App runs in Docker/Kubernetes Respect container CPU limit (cpu_limit = 2) → set workers = 2 🧠 App has high memory usage Use fewer workers, increase threads
- 24. How Many Workers? Scenario Recommendation 🖥 App runs on VM/bare metal Use 1 worker per available CPU core 📦 App runs in Docker/Kubernetes Respect container CPU limit (cpu_limit = 2) → set workers = 2 🧠 App has high memory usage Use fewer workers, increase threads 🧪 Unclear limits or mixed load Start with 2–4 workers and benchmark
- 25. How Many Threads? Factor Guideline 🔄 I/O-bound app Use more threads (e.g. 16–32) to handle concurrency
- 26. How Many Threads? Factor Guideline 🔄 I/O-bound app Use more threads (e.g. 16–32) to handle concurrency 🧮 CPU-bound app Fewer threads (e.g. 4–8) to avoid contention
- 27. How Many Threads? Factor Guideline 🔄 I/O-bound app Use more threads (e.g. 16–32) to handle concurrency 🧮 CPU-bound app Fewer threads (e.g. 4–8) to avoid contention 🧵 Low traffic env (dev/staging) Use something like threads 1, 4
- 29. Platform as a Service (PaaS) Example: Heroku, fly.io ✅ Easy to get started and Built-in autoscaling
- 30. Platform as a Service (PaaS) Example: Heroku, fly.io ✅ Easy to get started and Built-in autoscaling ❌ Cost to scale and limited control over infrastructure for advanced tuning
- 31. Infrastructure as a Service (IaaS) Example: AWS, GCP, Azure ✅ Full control (Instances, Auto Scaling, Load Balancers)
- 32. Infrastructure as a Service (IaaS) Example: AWS, GCP, Azure ✅ Full control (Instances, Auto Scaling, Load Balancers) ❌ High availability requires manual setup, fixed resource provisioning
- 33. Container Orchestration Example: Kubernetes (can run on AWS, GCP, Azure, or On-Premise) ✅ Portability and fine-grained control over resources(CPU, memory, limits per pod)
- 34. Container Orchestration Example: Kubernetes (can run on AWS, GCP, Azure, or On-Premise) ✅ Portability and fine-grained control over resources(CPU, memory, limits per pod) ❌ Steep learning curve — complex concepts (pods, services, volumes)
- 35. Kubernetes Scaling Strategies ↔ HPA (Horizontal Pod Autoscaler) → reacts to CPU/memory usage
- 36. Kubernetes Scaling Strategies ↔ HPA (Horizontal Pod Autoscaler) → reacts to CPU/memory usage ↕ VPA (Vertical Pod Autoscaler) → adjusts resource requests for individual pods
- 37. Kubernetes Scaling Strategies ↔ HPA (Horizontal Pod Autoscaler) → reacts to CPU/memory usage ↕ VPA (Vertical Pod Autoscaler) → adjusts resource requests for individual pods 🧱 Cluster Autoscaler → adds/removes nodes to accommodate workload
- 38. Horizontal Pod Autoscaling (HPA)
- 40. WHAT WE LEARNED
- 41. Common Pitfalls 🧩 Issue 🛠 Cause ⚠ Impact DB Pool Mismatch THREADS > pool in database.yml Connection timeouts and ActiveRecord errors
- 42. Common Pitfalls 🧩 Issue 🛠 Cause ⚠ Impact DB Pool Mismatch THREADS > pool in database.yml Connection timeouts and ActiveRecord errors Excessive Thread Count Threads set too high without real concurrency need Increased memory usage, thread contention, and degraded app performance
- 43. Common Pitfalls 🧩 Issue 🛠 Cause ⚠ Impact DB Pool Mismatch THREADS > pool in database.yml Connection timeouts and ActiveRecord errors Excessive Thread Count Threads set too high without real concurrency need Increased memory usage, thread contention, and degraded app performance Resource Overcommit Too many workers × threads for available CPU/memory Application instability due to memory exhaustion (OOM) or CPU throttling
- 44. Database Insights ➡ Read Replicas
- 45. Database Insights ➡ Read Replicas ➡ Add and maintain proper indexes
- 46. Database Insights ➡ Read Replicas ➡ Add and maintain proper indexes ➡ Optimize slow queries
- 47. Database Insights ➡ Read Replicas ➡ Add and maintain proper indexes ➡ Optimize slow queries ➡ Use partitioning
- 48. Database Insights ➡ Read Replicas ➡ Add and maintain proper indexes ➡ Optimize slow queries ➡ Use partitioning ➡ Consider cache strategies
- 49. Pro Tips Use background jobs Offload heavy work to async queues
- 50. Fail fast, retry smart Use retries with backoff (Sidekiq, Shoryuken, etc.) to avoid overload loops Pro Tips
- 51. Use observability tools Datadog, Sentry, NewRelic, AppSignal, Skylight… Pro Tips
- 52. Thank you! gustavoaraujo.dev garaujodev garaujodev We are hiring! cloudwalk.io/jobs