EUC2015 - Load testing XMPP servers with Plain Old Erlang
- 1. Tutorial: Load testing XMPP servers with Plain Old Erlang EUC 2015 tutorial, Stockholm Paweł Pikuła
- 2. Who am I? • Paweł Pikuła pawel.pikula@erlang-solutions.com • Developer @ Erlang Solutions Ltd. Cracow office • I work on MongooseIM XMPP server and some IoT related project. • github.com/ppikula • twitter: @pawelpikula
- 3. Agenda • What is load testing • Introducing AMOC - our simple load testing tool • Overview of our test infrastructure • Breaking a single node • Distributed scenario (scaling MIM and AMOC horizontally) • What to do next ?
- 4. Prepare the environment • Install VirtualBox and Vagrant • git clone https://github.com/ppikula/euc2015/ • Follow instructions from the README file (Setting up the environment from a usb stick)
- 5. What is load testing ? “In software engineering, performance testing is in general testing performed to determine how a system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.” wikipedia
- 6. Why do we do load test ? • Measure capacity of the whole system. • Measure latencies, time to deliver and other quality attributes. • Find the bottlenecks in our system. • Test a bad case scenarios under heavy load (ex. net splits, overloaded DBs). • Discover possible race conditions . • Stress 3rd party software.
- 7. Types of tests - endurance
- 8. Types of traffic - stress
- 9. Types of traffic - spike ! • Check if the system can recover after the spike • How long does it take to go back to the normal state
- 10. TSUNG • We‘ve been using it for a long time • It does the job • Tsung is dumb - doesn’t understand XMPP • XML DSL … • Manual setup • It is not designed for bidirectional protocols • HTTP semantics (request, transaction, page time) • some strange race conditions? (I was never able to login 10K users every time I tried I had ~9997 of them )
- 11. TSUNG DSL
- 12. OUR USE CASE • stream management enabled (XEP 198) auto respond to <r> with <a> • authentication process requires acquiring a token from http service • carbon copies - multiple jabber resources
- 13. ESCALUS • esl/ejabberd_tests use it for black box testing • supports many transports (ws, bosh, tcp, tls) • stanza generators • built-in predicates (is_message, is_iq_result)
- 14. Escalus story
- 15. Can we use escalus ?
- 16. AMOC design goals • Use the esl/escalus library • Don’t reinvent the wheel - cooperate with graphite, ansible, graphite notifications, graylog. • Simple and practical • No DSL - plain erlang • build everything on one host
- 17. AMOC features • Automatic deployment and environment setup via ansible • Exometer for collecting metrics - very easy to add a new metric • Uses modified erlang slave module • Dynamically increase/decrease the amount of users • Sends notifications about events mentioned above
- 18. Environment checklist • File descriptors limit • Check firewall rules • TCP buffers and other options • Increase local port range • Create more network interfaces if you want to generate more than 60K users
- 19. Why there is a limit of 60K per interface? • TCP port is a16 bit integer - we have only 65536 outgoing ports - reserved ones
- 20. AMOC Inside ! • AMOC scenario - define init and start function • AMOC controller - “user interface” start/stop scenarios. • The controller spawns users on AMOC slaves in round robin fashion
- 21. SUT - MongooseIM
- 22. MongooseIM • Fork of ejabberd 2.1.8 • Removed “non-scalable” extensions • Added unit & integration tests (black box testing) • Added many metrics • Want to hear more? go to “MongooseIM - The Right Tool for Scalable Messaging” by Michał Piotrowski
- 23. Task 1 break single box
- 24. Task 2 scale it !
- 25. Task3 Add custom metric to AMOC • Message rate (spiral) • Connected users count (counter) • Time to deliver (histogram)
- 26. Future of AMOC • automatic tests • allow to pass inter arrival per scenario • support for multiple scenarios • ansible recipes for ubuntu/debian family • dockerized version of AMOC? • integration with elixir - that will allow us to create DSLs
- 27. Thanks !
- 28. Questions ?