HAProxyConf 2019: Building a Service Mesh at Criteo with Consul and HAProxy
- 1. Pierre Souchay Twitter: @vizionr Github: pierresouchay Building a Service Mesh at Criteo with Consul and HAProxy Leading Auth/Discovery/Sec Team @Criteo Dealing with 270k+ services, 50k Consul nodes in 12 DCs 1st external contributor to Consul Author of consul-templaterb
- 2. Menu β’ Starters β’ Consul @Criteo β’ Main β’ Why Infrastructure Service Mesh? β’ Cheeses (many of them) β’ How we built HAProxy-Connect β’ Sweets β’ Features
- 3. 3 β’ Consul History at When, Why, How ?
- 4. 4 β’ More servers every year DC: 12 (9 prod) Servers: 50k+ Services: 4k+ Instances: 270k+ HTTP req/s: 7M+ BigData: 180+ Pb Kafka msg/s: 10M+ Criteo is a major Advertiser
- 5. 2013 - Hosts β’ Front-Facing LB β’ Web-Apps (few 100βs) β’ Micro-Services (few 100βs) β’ Many backends (few 100βs)
- 10. 2015 - Mesos β’ Containers β’ Frequent changes β’ Many services/machine β’ Different Provisioning
- 13. 1 2 3 4 Provisioning time is an issue globally (F5) Database polling shows its limits Services both in containers and machines? More latency Introduced By new Load-Balancers Sounds almost good enough butβ¦
- 14. Time to move on
- 15. What is Consul? β’ OSS Distributed Discovery system β’ Each agent declares its services β’ IP/PORT + health + TAGS + META β’ Anyone can discover all services β’ Notified when services changes β’ Provides DNS! β’ β¦ β’ Fault tolerance β’ Multi DC/Clouds / Legacy compatible
- 16. Step 1 Register * in Consul Deployer ΓΌAdd Health-Checks ΓΌAdd Tags
- 17. Step 2 : re- implement our libraries HTTP Client Side LB Database Access Kafka Memcached/Couchbase
- 20. Can we go further? Can we simplify?
- 21. The CSLB in our apps is hard to maintain β’ 10k lines of code in our SDKs β’ TLS is hard β’ 3 SDKs => 3 implementations
- 22. Consul Connect Hi-Level View A set of APIs in Consul Plug any LB Envoy by default
- 23. HAProxy-Connect β’ Open-Source β’ Built in go for static/easy deployment β’ Implementation of Consul Connect β’ Designed/Built by Criteo β’ Will be transferred to HAProxy Technologies!
- 24. HAProxy- Connect Configures fully HAProxy β’ No Need to create any HAProxy configuration β’ Listen all changes in Consul topology β’ TLS between apps for free! β’ Bare-Metal / Containers compatible
- 26. TLS Mutual TLS TLS provided by Consul for the client and target Client validates the server it is talking to Mutual authentication, server knows caller Client/Server have public CA of Consul, so they can validate each other Rotation is supported Transparent mechanism supported by Consul Can work with HashiCorpβs Vault transparently
- 28. At Service1 level βvery staticβ Frontend βmy_exposed_service_tlsβ β’ Config TLS β’ SPOE filter (will see later) Backend βlocalhostβ to service1 β’ Config TLS
- 29. At Service1 level βvery staticβ Frontend βmy_exposed_service_tlsβ β’ Config TLS β’ SPOE filter (will see later) Backend βlocalhostβ to service1 β’ Config TLS # Ingress ## Ingress frontend, handling TLS frontend front_downstream mode http bind 0.0.0.0:21000 name front_downstream_bind crt crt.pem ca-file ca.pem ssl verify required # Configure SPOE filter spoe engine intentions config spoe.conf # Reject based on SPOE response tcp-request content reject unless { var(sess.connect.auth) -m int eq 1 } default_backend back_downstream ## Backend in plain text backend back_downstream mode http server downstream_node 127.0.0.1:31415
- 30. At App1 level βvery dynamicβ Frontend βservice1_target_plainβ β’ Listen on plain text to localhost Backend βservice1_targets_tlsβ to service1 β’ Config TLS client β’ Validates service1_TLS using CA β’ service1_instance000 β’ service1_instance001 β’ β¦
- 31. At App1 level βvery dynamicβ Frontend βservice1_target_plainβ β’ Listen on plain text to localhost Backend βservice1_targets_tlsβ to service1 β’ Config TLS client β’ Validates service1_TLS using CA β’ service1_instance000 β’ service1_instance001 β’ β¦ # Egress to service1 ## Frontend listening on localhost, in plaintext frontend front_service1 mode http bind 127.0.0.1:31417 name front_service1_bind timeout client 30000 default_backend back_service1 ## Backend handling TLC backend back_service1 mode http balance leastconn timeout server 60000 timeout connect 1000 server srv_0 10.0.0.100:21000 enabled ssl weight 1 crt crt.pem ca-file ca.pem server srv_1 10.0.0.101:21000 enabled ssl weight 1 crt crt.pem ca-file ca.pem server srv_2 10.0.0.102:21000 enabled ssl weight 1 crt crt.pem ca-file ca.pem server srv_3 10.0.0.103:21000 enabled ssl weight 1 crt crt.pem ca-file ca.pem
- 32. Service1 can also have targets
- 34. Intentions
- 35. Implementation of Intentions HAProxy-connect also listen to SPOE requests HAProxy can thus validate if appX can target service You can get a global graph of calls across Β΅Services!
- 37. What is SPOE HAProxy-Connect Configuration SPOE Send TLS Client Certificate for validation
- 38. HAProxy-Connect as SPOA β’ Decodes TLS certificate (SPIFFE) β’ Extract name of caller (Consul Service) β’ Call Consul agent with service name to validate intention β’ If not OK => Connection is closed
- 39. Under the hood β’ HAProxy-Connect 1. Generates initial HAProxy configuration (Basic conf with ctrl socket + SPOE) 2. Starts HAProxy 3. Starts dataplane-api with generated config β’ Uses dataplane-API β’ Rest API to control HAProxy configuration dynamically β’ Removes complexity of socat & friends
- 40. Optimizations β’ Reload is evil, we over allocate per power of 2 slots β’ HAProxy-Connect enable/disable slots according to their state Slot of servers overallocated β’ HAProxy-Connect applies configuration changes β’ Watch the global result β’ If different, re-apply all the changes until convergence works => can work if someone else is playing with us Convergence
- 41. Statistics β’ Minimalist but efficient HAProxy stats page β’ For EGRESS β with full details β’ For INGRESS Prometheus
- 42. Statistics EGRESS (App1) β Per target/global
- 44. Logs HAProxy- Connect collects all of them Can be redirected to syslog/system if you want
- 45. Cool stuff / QA β’ Open-Source! β’ https://github.com/criteo/HAProxy-consul-connect β’ Implemented SPOP to implement SPOA for Golang β’ https://github.com/criteo/HAProxy-spoe-go β’ Some others can benefit too! β’ https://github.com/clems4ever/HAProxy-ldap-auth β’ Thanks to https://github.com/Aestek @vizionr Github: pierresouchay