Scaling Runa Inc Big Data e-commerce service with AWS
1 like1,176 views
Runa is a SaaS startup that enhances online commerce by delivering personalized promotions to shoppers in real-time using AWS for scalable big data and machine intelligence solutions. The company faced technical challenges in processing large volumes of data as it aimed to support thousands of merchants, employing various technologies like Hadoop, HBase, and Clojure for performance and reliability. AWS was selected for its flexibility, cost-effectiveness, and ability to scale while minimizing upfront investment, allowing Runa to rapidly prototype and grow their services.
Scaling Runa Inc Big Data e-commerce service with AWS
- 1. Runa on AWS Big Data & Machine Intelligence for a SaaS Startup
- 2. Runa
- 3. a SaaS
- 8. in Real-Time
- 10. Tech Challenges
- 11. Big Data
- 12. JavaScript client collects activity on every Merchant page for every Shopper
- 13. One or more Ajax call & Event Store to Runa per Merchant page view
- 14. Step function increase of calls and stores as each new Merchant added
- 15. We capture everything we can and store it forever
- 16. Expecting to grow to thousands of merchants
- 17. That’s a lot of Data
- 19. Batch Processing for Statistical Analysis and Reports
- 22. At First (a couple years ago)
- 23. Not Much Money in the Bank
- 25. Or exactly how we were going to do it
- 26. Prototyped with Ruby / Rails / MySQL
- 28. EC2 & AWS let us scale the prototype to Beta Production
- 29. Flexibility to incrementally refine service & infrastructure
- 30. Confidence we could scale as we added Merchants
- 31. More Recently Incrementally added next-gen Tech & Full Production
- 33. Batch Processing & Infinite Storage Map / Reduce & BigTable via Hadoop & HBase
- 34. Flexible Real-Time parallel processing via Clojure / Swarmiji
- 35. Opscode Chef Management & Monitoring Consumers on Merchant Websites Internet Admin & Merchant Dashboard (Rails) Runtime Rules Merchant Info Merchants Internet AnalyticsReporting Monitor & Recovery Data Collectors Hadoop / HBase Map / Reduce Petabyte Store Load Balancer HTTP Shared Session Memory HTTP Dispatchers Redis Mem Cache Redis Mem Cache Redis Mem Cache Redis Mem Cache Redis Mem Cache HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase Amazon S3 Data Backup 9+ Amazon EC2 Instances Amazon Elastic Load Balancer 3+ Amazon EC2 Instances Cheshire / Swarmiji Dynamic Runtime Queue
- 36. AWS Elastic Load Balancer Rails App Servers Nginx / Unicorn EC2 m1.xlarge MySQL Master / Slave EC2 m1.xlarge EBS Legacy Runtime Rails App Nginx/Unicorn MySQL Master/Slave EC2 m1.xlarge / EBS Merchant Dashboard EC2 m1.xlarge HBase / Hadoop EC2 m1.xlargeEBS RabbitMQ Cheshire / Swarmiji Redis EC2 m1.xlarge Clojure Based Runtime AWS Elastic Load Balancer EC2 m1.large Opscode Chef Monitoring EC2 m1.large All Deployed on
- 38. Good Things
- 39. Able to Start Small
- 40. Then GROW BIGGER
- 41. Having the flexibility to throw “Hardware” at our Prototype got us to market faster
- 42. Ability to launch test and staging environments almost at will
- 46. Managing Complexity lots of moving parts
- 47. Easy to launch a few instances
- 49. Must have tool like Opscode Chef
- 50. Chef automates deployment & puts it under Revision Control
- 51. There’s going to be some blood when using cutting edge tech
- 54. HBase on AWS may be dangerous because of Hadoop namenode SPOF
- 55. EC2 bill can surprise you if you cavalierly deploy multiple versions of horizontally scalable environments
- 56. Could not do our startup without AWS or lots more VC Funding