![Circuit Breaker for remote Services
Pattern is used to detect failures and encapsulates logic of
preventing a failure to reoccur constantly[1]
If a service instance throws an error, times out, or responds
with a failure message an error event is marked
If the error rate threshold is exceeded that service instance is
removed from the pool of available services
Before re-adding a service to the pool a test request is made
and validated
Internal service failures should not be reflected in response to
message originator
[1] - http://en.wikipedia.org/wiki/Circuit_breaker_design_pattern](https://image.slidesharecdn.com/bigdatadcskyfallpresov2-121212101931-phpapp02/85/Big-datadc-skyfall_preso_v2-17-320.jpg)
Big datadc skyfall_preso_v2
- 1. Project Skyfall Matt Abrams (@abramsm)
- 2. Agenda A bit about AddThis! ! Why did we need Skyfall?! ! Architecture! ! Operations/Performance!
- 5. Fun with Numbers AddThis JavaScript loads > 3 Billion times per day Edge Network (Skyfall) receives around 4B hits per day Either datacenter can handle 100% load (we test this often) Currently using around 1K servers (will double next year)
- 7. Why did we need Skyfall? We couldn’t find anyone else to do it for us • Pervious vendors log aggregation was delayed by a minimum of 3 hours and could take up to 5 days Minimize impact on our publishers • Combining log collection with remote services means we only need 1 event instead of n Support near real time applications
- 8. Why did we call it Skyfall?
- 9. Why did we call it Skyfall?
- 10. Skyfall Goals and Architecture!
- 11. Skyfall Goals (Technical) High Availability Handle Server and DC failure gracefully Low latency Zero downtime deployment and configuration Use for internal and external Logging needs In session RPC O(1) reads and writes Support data filtering at the edge Smart Clients
- 12. Why speed and robustness matters
- 13. Architecture Web Event Web Event Web Event Global Traffic Management DC1 DC2 Skyfall Skyfall Skyfall Skyfall Skyfall Skyfall Repeater Consumer Service Consumer Consumer Consumer Service Service Consumer Service Consumer Consumer Service Consumer Service
- 15. 1. Messages are placed on concurrent non-blocking queue (CNBQ) to minimize latency impact on producer 2. Messages are then popped from CNBQ and placed on a Disk-Backed queue (DBQ) 3. DBQ is used to provide temporary storage in case Kafka is down or backed up 4. Messages from DBQ are popped and sent to Kafka where they are persisted to file system
- 16. Kafka Kafka is treats persistence as a first class citizen Focus is on high throughput vs lots of bells and whistles State about what has been consumed is maintained in the client rather than the server Kafka is explicitly distributed Supports O(1) reads and writes Pull rather than push http://incubator.apache.org/kafka/design.html
- 17. Circuit Breaker for remote Services Pattern is used to detect failures and encapsulates logic of preventing a failure to reoccur constantly[1] If a service instance throws an error, times out, or responds with a failure message an error event is marked If the error rate threshold is exceeded that service instance is removed from the pool of available services Before re-adding a service to the pool a test request is made and validated Internal service failures should not be reflected in response to message originator [1] - http://en.wikipedia.org/wiki/Circuit_breaker_design_pattern
- 18. What does a call to our endpoint look like? Topic • "GET /live/t00/250lo.gif&foo=bar" 200 37 "http:// s7.addthis.com/static/r07/sh103.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/ 5.0)"!
- 19. What does a call to our endpoint look like? Version Topic • "GET /live/t00/250lo.gif&foo=bar" 200 37 "http:// s7.addthis.com/static/r07/sh103.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/ 5.0)"!
- 20. What does a call to our endpoint look like? Version Resource Topic • "GET /live/t00/250lo.gif&foo=bar" 200 37 "http:// s7.addthis.com/static/r07/sh103.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/ 5.0)"!
- 21. What does a call to our endpoint look like? Version Resource URL Params Topic • "GET /live/t00/250lo.gif&foo=bar" 200 37 "http:// s7.addthis.com/static/r07/sh103.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/ 5.0)"!
- 22. What does a call to our endpoint look like? Version Resource URL Params Status Code Topic • "GET /live/t00/250lo.gif&foo=bar" 200 37 "http:// s7.addthis.com/static/r07/sh103.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/ 5.0)"!
- 23. What does a call to our endpoint look like? Version Resource URL Params Status Code Topic Bytes Transferred • "GET /live/t00/250lo.gif&foo=bar" 200 37 "http:// s7.addthis.com/static/r07/sh103.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/ 5.0)"!
- 24. What does a call to our endpoint look like? Version Resource URL Params Status Code Topic Bytes Transferred • "GET /live/t00/250lo.gif&foo=bar" 200 37 "http:// s7.addthis.com/static/r07/sh103.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/ 5.0)"! CDN Resource User Agent
- 25. What does a call to our endpoint look like? Version Resource URL Parameters Status Code Topic Bytes Transferred "GET /live/t00/250lo.gif&foo=bar" 200 37 "http:// s7.addthis.com/static/r07/sh103.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/ 5.0)" CDN Resource User Agent The endpoint also receives header and cookie information not Shown here.
- 26. Zero Downtime Deployment and Configuration Group 1 4 8 16 S1 S2 S2 S3 S3 S4 S4 S5 S5 Group 2 4 8 16 S1 S2 S2 S3 S3 S4 S4 S5 S5
- 27. Endpoint Configuration Each endpoint maps to a ‘topic’ Header elements may be extracted from the HTTP request Parameters may be mapped to new key names Variables may be extracted from the URL path
- 28. Data Center Repeater DC Repeater nodes automatically negotiate N1 peering relationships with nodes in the other data N1 center N2 If a peer node becomes unreachable the local node N2 will select a new peer N3 These are special consumers of the Kafka log data created by the local node
- 31. Requests per/second (VA Data Center)
- 32. TCP - When do you say goodbye? http://upload.wikimedia.org/wikipedia/commons/a/a2/Tcp_state_diagram_fixed.svg
- 33. Connection Tracking – what you need to know Connection information is maintained in memory The message: “ip_conntrack: table full, dropping packet” is BAD Chrome – doesn’t close connection on FIN This means that the connection info remains open until it times out, drastically increasing the number of connection your server needs to track You need some mechanism for timing out the connection in a reasonable time period
- 34. HA Proxy We use a simple round-robin load balancing algorithm with a liveness check Default connection timeouts are way to high. Reasonable values are used to prevent excessive connection tracking “http-close” and “http-server-close” are enabled to ensure low latency for clients and fast session reuse for the server HA Proxy is our solution of choice our LB needs. We prefer software solutions on commodity hardware vs expensive custom LB appliances They could use a new logo