Service Discovery in Distributed Systems
7 likes5,449 views
Service discovery allows services in distributed systems to find and connect with each other. Popular open-source service discovery solutions include ZooKeeper, Eureka, ETCD, and Consul. ZooKeeper uses consensus protocol ZAB for consistency but clients must handle load balancing. Eureka is eventually consistent and Netflix uses it for AWS. ETCD is inspired by ZooKeeper but clients also handle load balancing. Consul uses RAFT consensus and has a built-in DNS server, making it a comprehensive solution. The right service discovery solution depends on requirements like consistency needs and language integration.
![Review of Open Source Projects
Project Name Implementor Год рождения
Chubby Google 2006
ZooKeeper [*] Apache 2007
Doozerd Blake Mizerany 2010
My in-house solution Deutsche bank 2012
Eureka [*] Netflix 2012
ETCD [*] CoreOS 2013
SmartStack AirBnb 2013
Surf HashiCorp 2014
SkyDNS Erik St. Martin 2014
Consul [*] HashiCorp 2014](https://image.slidesharecdn.com/servicedicsoveryindistributedsystems-150304130959-conversion-gate01/85/Service-Discovery-in-Distributed-Systems-11-320.jpg)
Service Discovery in Distributed Systems
- 1. Service Discovery In Distributed Systems IVAN VOROSHILIN @vibneiro
- 2. My Interests: Ivan Voroshilin - Computer geek Bio Distributed systems Architecture Functional languages Algorithms
- 4. OMG! This is an unmanageable mess! Google Twitter Netflix Facebook
- 5. Why? I don’t have to restart a service to update configuration I don’t have to redeploy something, when a service is moved to another box I don’t have to know whether a needed service is alive at the moment Load-balancing might be taken care for you by Service Discovery also Service Discovery takes a responsibility to store configs for you in 1 place
- 6. Agenda - Main properties of Service Discovery - Popular Open-Source Solutions Differences, Trade-offs - Bonus
- 7. Service Discovery The main components
- 8. The main idea of Service Discovery in simple terms Announce/LookUp Service Directory Register Services Discover Services Consistent Highly-available a.k.a. Announce a.k.a. LookUp
- 9. CAP? ConsistencyAvailability PartitionTolerance 2 out of 3 properties: Either CP or AP. No other options
- 10. Issues when developing Service Discovery Fault Tolerance Data Consistency Distributed Locks Leader Election (Itisnottrivial!)
- 11. Review of Open Source Projects Project Name Implementor Год рождения Chubby Google 2006 ZooKeeper [*] Apache 2007 Doozerd Blake Mizerany 2010 My in-house solution Deutsche bank 2012 Eureka [*] Netflix 2012 ETCD [*] CoreOS 2013 SmartStack AirBnb 2013 Surf HashiCorp 2014 SkyDNS Erik St. Martin 2014 Consul [*] HashiCorp 2014
- 12. Name resolution and DNS Announce/Lookup?! Sounds like DNS.. - Scales out badly - Split-brain - Too simple
- 14. Service Discovery Open Source Solutions
- 15. 2 Categories of solutions - General purpose solutions: ZooKeeper, ETCD - Solely Service Discovery solutions: Eureka, Surf, Consul
- 16. Apache Zookeeper - Consensus protocol ZAB, CP - Written in Java - Language binding: Java and C API - Key-value store based on ephemeral nodes - Clients need to handle any load balancing or failover themselves - On any non-quorum side, reads and writes will return an error.
- 17. Apache Zookeeper In Hadoop, Mesos, Kafka, Netflix Distributed lock for coordination - Heavyweight - Difficult to use - Not ideal for Service Discovery - Ephemeral nodes not reliable
- 18. Netflix Eureka - Eventually consistent, AP - Broadcast async replication among servers, no quorum - HTTP-REST API - Written in Java - Java smart client with round robin balancer - Caching of server entries on the client side - add/remove nodes online - TTL refreshment mechanism - Used for AWS cloud
- 19. ETCD - Inspired by Zookeeper - Consensus protocol - RAFT, CP - Written in Go - HTTP REST API - Key-value: Store data in directories - TTLs for keys expiration - Clients need to handle any load balancing or failover themselves - Watch a key or directory for changes and react to the new values
- 20. Consul - Consensus protocol RAFT: CP - HTTP REST API or even DNS interface - Built-in DNS server as a service registry - Security: TLS, ACLs - Comprehensive solution for Service Discovery - Specialized health-checks, besides just heartbeats - Optional read consistency mode: stale (when leader is unavailable)
- 21. What should do a comprehensive Service Discovery? Nothing from the mentioned answers the following question: How to without a hassle to integrate your existing landscape with a service discovery solution? Doesn’t matter whether 3-party or in-house applications What about closed 3-rd party systems? Language binding? Need to write client code to adapt?
- 22. Service Discovery in Docker environment Meet - “Side-kick” processes. (Introduced in SmartStack and Serf Solutions) Client application Backend 1 Backend 2 application HAProxy ETCD-cluster service process side-kick service process side-kick discover HTTP/TCP proxied HTTP/TCP proxied announce announce watchhealth check TTL TTL check health
- 23. Summary No One size fits all solutions The right architecture or Open-Source solution directly depends on clear requirements
- 24. Thanks for attention blog: www.ivoroshilin.com Q & A