SlideShare a Scribd company logo
WebSocket introspection
and stress testing with
TCPKali
by Lev Walkin
<lwalkin@mz.com>
@levwalkin
WebSocket
• Persistent, bi-directional message layer on top of TCP
• Compact binary framing (2-14 bytes overhead)
• Extensible (compression, custom protocols
negotiation)
• Rather simple in its basic form, but TCP-only tooling
does not work since WS starts with HTTP req/resp.
• Native in modern browsers, Secure wss:// is rather
compatible with old proxies and network devices.
MZ (Machine Zone)
• Game of War and Mobile Strike
• New RTplatform™
• MZ’s real time platform is a competitive asset
• (Don’t let embedded systems engineers
appropriate the “real-time” term)
MZ Challenges
• Support hundreds of millions events per second to
hundreds of millions of global users and devices
• Large scale TCP-only load testing and introspection
tools do not play well with WebSockets
• Answer: tcpkali (fast, single-node, CLI)

and MZBench (thorough, distributed, Web UI)
• MZBench talk by Vibhav Garg:

https://www.youtube.com/watch?v=ravA5emK9UE
TCPKali
• TCP and WebSocket
• Large scale persistence
• Extremely fast
• Developer-friendly
• https://github.com/
machinezone/tcpkali
• brew install tcpkali
Large Scale Systems
Developer Challenges
• Stress-testing breaks things.
• Many clients break things.
• Slow clients break things.
• Some WebSocket framing options break things.
• Need detailed visibility into what’s going on.
Stress-testing
• Idea: repeat message many times
• tcpkali --ws 127.1:8080 -m ‘MyData’
• tcpkali --ws 127.1:8080 -d # dump
• tcpkali --ws 127.1:8080 -m ‘MyData’ -d -r100
• tcpkali -l 8080 127.1:8080 -m z # self-test
Breaking Node.JS
• brew install tcpkali
• brew install npm && npm install websocket
• Grab code from http://cjihrig.com/blog/websockets-in-
node-js-0-8-6-for-windows-7/ remove ‘echo-protocol’.
• node websocket-server.sh
• tcpkali --ws -mz 127.1:8080
• Observe `top -ocpu`…
Many clients
• Idea: establish many mostly idle connections
• tcpkali 127.1:8080 --websocket --message-rate 0.1

--message ‘{ws.ping}’ --duration 1h

--connections 1m
• tcpkali --ws -r0.1 -m ‘{ws.ping}’ -T1h --c1m 127.1:8080
• Requires ~=1m/64k interface aliases on a host.
• tcpkali automatically picks up all interface aliases.
• tcpkali will suggest tuning kernel options!
Slow clients
• Idea: establish bandwidth limits
• tcpkali --ws 127.1:8080 -m ‘message’

--channel-bandwidth-downstream 14.4kbps
• Some thing in the opposite direction:

--channel-bandwidth-upstream 42mbps
WebSocket Framing
• Idea: easy construction of non-trivial framing
• … -m ‘here: {ws.binary </etc/passwd>}’
• … -m ‘{ws.continuation “unexpected frame”}’
• … -1 ‘{ws.text “Mary” …}’ -m ‘{ws.cont “had”}’
• (the last one may trigger infinite memory
consumption on a peer prior to receiving a
message by application due to frame reassembly)
Developer Visibility
• `-d` dumps some random connection’s TCP-level I/O,
for casual observation
• --dump-all-in, --dump-all-out, --dump-one-in --dump-
one-out will help to get more or less data.
• --latency-connect, --latency-first-byte, --latency-
marker for protocol and application level SLA checks
• --statsd exports many internal metrics to graph
progress (via e.g. DataDog)
2016-tcpkali-websocket
What it isn’t
• Not a security testing tool. Use Burp Suite.
• Not a distributed testing tool. Use MZBench.
• Not a scenario based testing. Use MZBench
• Though RegExes like “{re [a-z]+}” can be
used to add some variety to message data.
Source and Documentation
• Requires bison and flex (brew install, yum install, …)
• git clone https://github.com/machinezone/tcpkali &&
cd tcpkali && autoreconf -iv && ./congure && sudo
make install
• tcpkali -h # most useful options
• tcpkali --help # full list of command line options
• man tcpkali # more thorough help
Lev Walkin @levwalkin
Q&A

More Related Content

PPTX
TCP Issues in DataCenter Networks
PDF
Zmq in context of openstack
PDF
Rails Conf Europe 2007 - Utilizing Amazon S3 and EC2 in Rails
PPTX
Rabbit MQ
PDF
NullMQ @ PDX
PPTX
zeromq
PDF
Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 Instance
PDF
OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...
TCP Issues in DataCenter Networks
Zmq in context of openstack
Rails Conf Europe 2007 - Utilizing Amazon S3 and EC2 in Rails
Rabbit MQ
NullMQ @ PDX
zeromq
Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 Instance
OSv Unikernel — Optimizing Guest OS to Run Stateless and Serverless Apps in t...

What's hot (20)

PDF
Introduction to ZeroMQ - eSpace TechTalk
PDF
Rails hosting
 
PDF
ZeroMQ with NodeJS
KEY
ZeroMQ
PPT
ruby + websocket + haproxy
PPT
Leveraging zeromq for node.js
PDF
XMPP & AMQP
PDF
The Future of Messaging: RabbitMQ and AMQP
PPTX
The RabbitMQ Message Broker
PDF
MEAN-stack based sensor gateway
 
PDF
Messaging Standards and Systems - AMQP & RabbitMQ
PPTX
Message Broker System and RabbitMQ
PDF
Messaging with amqp and rabbitmq
KEY
Distributed app development with nodejs and zeromq
PDF
Full Stack Bus with Javascript, RabbitMQ and Postal.js
PDF
Half a year with contrail at production
PDF
Lindsay distributed geventzmq
PDF
Introduction to AMQP Messaging with RabbitMQ
PDF
http server on user-level mTCP stack accelerated by DPDK
 
PDF
[@NaukriEngineering] Messaging Queues
Introduction to ZeroMQ - eSpace TechTalk
Rails hosting
 
ZeroMQ with NodeJS
ZeroMQ
ruby + websocket + haproxy
Leveraging zeromq for node.js
XMPP & AMQP
The Future of Messaging: RabbitMQ and AMQP
The RabbitMQ Message Broker
MEAN-stack based sensor gateway
 
Messaging Standards and Systems - AMQP & RabbitMQ
Message Broker System and RabbitMQ
Messaging with amqp and rabbitmq
Distributed app development with nodejs and zeromq
Full Stack Bus with Javascript, RabbitMQ and Postal.js
Half a year with contrail at production
Lindsay distributed geventzmq
Introduction to AMQP Messaging with RabbitMQ
http server on user-level mTCP stack accelerated by DPDK
 
[@NaukriEngineering] Messaging Queues
Ad

Similar to 2016-tcpkali-websocket (20)

PDF
Tuning the Kernel for Varnish Cache
PDF
10 sdn-vir-6up
PDF
Memory, Big Data, NoSQL and Virtualization
PDF
All Bow To OpenSolaris Crossbow
PDF
All Bow To Open Solaris Crossbow H4ck3rz Due
PDF
SR-IOV: The Key Enabling Technology for Fully Virtualized HPC Clusters
PPSX
MQTC V2.0.1.3 - WMQ & TCP Buffers – Size DOES Matter! (pps)
PPTX
Lecture notes - Data Centers________.pptx
PDF
What every Java developer should know about network?
PDF
Anton Moldovan "Building an efficient replication system for thousands of ter...
 
PDF
Design and Performance Characteristics of Tap-as-a-Service
PDF
XPDS13: On Paravirualizing TCP - Congestion Control on Xen VMs - Luwei Cheng,...
PDF
Demystfying container-networking
PPTX
Quantum for Cloud Operators - Folsom Conference
PPTX
Installing Oracle Database on LDOM
PPT
Rabbit MQ introduction
PPTX
DevOops - Lessons Learned from an OpenStack Network Architect
PPTX
IP Multicast on ec2
PDF
Ubuntu server wireless access point (eng)
PDF
mTCP使ってみた
Tuning the Kernel for Varnish Cache
10 sdn-vir-6up
Memory, Big Data, NoSQL and Virtualization
All Bow To OpenSolaris Crossbow
All Bow To Open Solaris Crossbow H4ck3rz Due
SR-IOV: The Key Enabling Technology for Fully Virtualized HPC Clusters
MQTC V2.0.1.3 - WMQ & TCP Buffers – Size DOES Matter! (pps)
Lecture notes - Data Centers________.pptx
What every Java developer should know about network?
Anton Moldovan "Building an efficient replication system for thousands of ter...
 
Design and Performance Characteristics of Tap-as-a-Service
XPDS13: On Paravirualizing TCP - Congestion Control on Xen VMs - Luwei Cheng,...
Demystfying container-networking
Quantum for Cloud Operators - Folsom Conference
Installing Oracle Database on LDOM
Rabbit MQ introduction
DevOops - Lessons Learned from an OpenStack Network Architect
IP Multicast on ec2
Ubuntu server wireless access point (eng)
mTCP使ってみた
Ad

2016-tcpkali-websocket

  • 1. WebSocket introspection and stress testing with TCPKali by Lev Walkin <lwalkin@mz.com> @levwalkin
  • 2. WebSocket • Persistent, bi-directional message layer on top of TCP • Compact binary framing (2-14 bytes overhead) • Extensible (compression, custom protocols negotiation) • Rather simple in its basic form, but TCP-only tooling does not work since WS starts with HTTP req/resp. • Native in modern browsers, Secure wss:// is rather compatible with old proxies and network devices.
  • 3. MZ (Machine Zone) • Game of War and Mobile Strike • New RTplatform™ • MZ’s real time platform is a competitive asset • (Don’t let embedded systems engineers appropriate the “real-time” term)
  • 4. MZ Challenges • Support hundreds of millions events per second to hundreds of millions of global users and devices • Large scale TCP-only load testing and introspection tools do not play well with WebSockets • Answer: tcpkali (fast, single-node, CLI)
 and MZBench (thorough, distributed, Web UI) • MZBench talk by Vibhav Garg:
 https://www.youtube.com/watch?v=ravA5emK9UE
  • 5. TCPKali • TCP and WebSocket • Large scale persistence • Extremely fast • Developer-friendly • https://github.com/ machinezone/tcpkali • brew install tcpkali
  • 6. Large Scale Systems Developer Challenges • Stress-testing breaks things. • Many clients break things. • Slow clients break things. • Some WebSocket framing options break things. • Need detailed visibility into what’s going on.
  • 7. Stress-testing • Idea: repeat message many times • tcpkali --ws 127.1:8080 -m ‘MyData’ • tcpkali --ws 127.1:8080 -d # dump • tcpkali --ws 127.1:8080 -m ‘MyData’ -d -r100 • tcpkali -l 8080 127.1:8080 -m z # self-test
  • 8. Breaking Node.JS • brew install tcpkali • brew install npm && npm install websocket • Grab code from http://cjihrig.com/blog/websockets-in- node-js-0-8-6-for-windows-7/ remove ‘echo-protocol’. • node websocket-server.sh • tcpkali --ws -mz 127.1:8080 • Observe `top -ocpu`…
  • 9. Many clients • Idea: establish many mostly idle connections • tcpkali 127.1:8080 --websocket --message-rate 0.1
 --message ‘{ws.ping}’ --duration 1h
 --connections 1m • tcpkali --ws -r0.1 -m ‘{ws.ping}’ -T1h --c1m 127.1:8080 • Requires ~=1m/64k interface aliases on a host. • tcpkali automatically picks up all interface aliases. • tcpkali will suggest tuning kernel options!
  • 10. Slow clients • Idea: establish bandwidth limits • tcpkali --ws 127.1:8080 -m ‘message’
 --channel-bandwidth-downstream 14.4kbps • Some thing in the opposite direction:
 --channel-bandwidth-upstream 42mbps
  • 11. WebSocket Framing • Idea: easy construction of non-trivial framing • … -m ‘here: {ws.binary </etc/passwd>}’ • … -m ‘{ws.continuation “unexpected frame”}’ • … -1 ‘{ws.text “Mary” …}’ -m ‘{ws.cont “had”}’ • (the last one may trigger innite memory consumption on a peer prior to receiving a message by application due to frame reassembly)
  • 12. Developer Visibility • `-d` dumps some random connection’s TCP-level I/O, for casual observation • --dump-all-in, --dump-all-out, --dump-one-in --dump- one-out will help to get more or less data. • --latency-connect, --latency-rst-byte, --latency- marker for protocol and application level SLA checks • --statsd exports many internal metrics to graph progress (via e.g. DataDog)
  • 14. What it isn’t • Not a security testing tool. Use Burp Suite. • Not a distributed testing tool. Use MZBench. • Not a scenario based testing. Use MZBench • Though RegExes like “{re [a-z]+}” can be used to add some variety to message data.
  • 15. Source and Documentation • Requires bison and flex (brew install, yum install, …) • git clone https://github.com/machinezone/tcpkali && cd tcpkali && autoreconf -iv && ./congure && sudo make install • tcpkali -h # most useful options • tcpkali --help # full list of command line options • man tcpkali # more thorough help