![[DO16] Mesosphere : Microservices meet Fast Data on Azure](https://image.slidesharecdn.com/do16-170616021542/85/DO16-Mesosphere-Microservices-meet-Fast-Data-on-Azure-3-320.jpg)
![© 2016 Mesosphere, Inc. All Rights Reserved. 20
Use: The primary resource manager and
negotiator
Why Mesos?
● 2-level scheduling
● Fault-tolerant, battle-tested
● Scalable to 10,000+ nodes
● Created by Mesosphere founder @ UC
Berkeley; used in production by 100+ web-
scale companies [1]
[1] http://mesos.apache.org/documentation/latest/powered-by-mesos/
APACHE MESOS](https://image.slidesharecdn.com/do16-170616021542/85/DO16-Mesosphere-Microservices-meet-Fast-Data-on-Azure-20-320.jpg)
[DO16] Mesosphere : Microservices meet Fast Data on Azure
- 1. SMACK Microservices meet Fast Data on Azure @joerg_schad
- 2. © 2017 Mesosphere, Inc. All Rights Reserved. 2 Jörg Schad Distributed Systems Engineer @joerg_schad
- 4. Video
- 5. © 2017 Mesosphere, Inc. All Rights Reserved. Hardware Operating System Application 5 Applications
- 6. © 2017 Mesosphere, Inc. All Rights Reserved. noun | ˈmīkrō/ /ˈsərvəs/ : an approach to application development in which a large application is built as a suite of modular services. Each module supports a specific business goal and uses a simple, well-defined interface to communicate with other modules.* Microservices are designed to be flexible, resilient, efficient, robust, and individually scalable. *From whatis.com OVERVIEW
- 7. © 2017 Mesosphere, Inc. All Rights Reserved. Operating System Operating System Operating System ServiceApp ServiceServiceAppApp 7 MICROSERVICES Polyglot Single Responsibility Smaller Teams Utilization Machine types/groups Dependency hell Machine Infrastructure Machine Machine ServiceService ServiceServiceServiceService
- 8. © Gerard Julien/AFP Run everything in containers!
- 9. © 2017 Mesosphere, Inc. All Rights Reserved. Container Internals
- 10. © 2017 Mesosphere, Inc. All Rights Reserved. Container Internals
- 11. © 2017 Mesosphere, Inc. All Rights Reserved. 11 MapReduce is crunching Data Meanwhile...
- 12. © 2016 Mesosphere, Inc. All Rights Reserved. 12 But then business demanded FAST DATA We need to turn faster! Today...
- 13. © 2016 Mesosphere, Inc. All Rights Reserved. 13 Fast Data Batch Event ProcessingMicro-Batch Days Hours Minutes Seconds Microseconds Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations
- 14. © 2016 Mesosphere, Inc. All Rights Reserved. 14 The SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Mesos/ DC/OS Sensors Devices Clients
- 15. © 2017 Mesosphere, Inc. All Rights Reserved. 15 Datacenter
- 16. © 2017 Mesosphere, Inc. All Rights Reserved. 16 NAIVE APPROACH Typical Datacenter siloed, over-provisioned servers, low utilization Industry Average 12-15% utilization mySQL microservice Cassandra Spark/Hadoop Kafka
- 17. © 2017 Mesosphere, Inc. All Rights Reserved. 17
- 18. © 2017 Mesosphere, Inc. All Rights Reserved. 18 MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS Typical Datacenter siloed, over-provisioned servers, low utilization Mesos/ DC/OS automated schedulers, workload multiplexing onto the same machines mySQL microservice Cassandra Spark/Hadoop Kafka
- 19. © 2017 Mesosphere, Inc. All Rights Reserved. • A top-level Apache project • A cluster resource negotiator • Scalable to 10,000s of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support 19 Apache Mesos
- 20. © 2016 Mesosphere, Inc. All Rights Reserved. 20 Use: The primary resource manager and negotiator Why Mesos? ● 2-level scheduling ● Fault-tolerant, battle-tested ● Scalable to 10,000+ nodes ● Created by Mesosphere founder @ UC Berkeley; used in production by 100+ web- scale companies [1] [1] http://mesos.apache.org/documentation/latest/powered-by-mesos/ APACHE MESOS
- 21. © 2016 Mesosphere, Inc. All Rights Reserved. 21 MESOS: FUNDAMENTAL ARCHITECTURE Mesos Master Mesos Master Mesos Master Mesos AgentMesos Agent Service Cassandra Executor Cassandra Task Cassandra Scheduler Container Scheduler Spark Scheduler Spark Executor Spark Task Mesos AgentMesos Agent Service Docker Executor Docker Task Spark Executor Spark Task Two-level Scheduling 1. Agents advertise resources to Master 2. Master offers resources to Framework 3. Framework rejects / uses resources 4. Agent reports task status to Master
- 22. © 2017 Mesosphere, Inc. All Rights Reserved. 22
- 23. Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) DC/OS ENABLES MODERN DISTRIBUTED APPS Big Data + Analytics EnginesMicroservices (in containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Any Infrastructure (Physical, Virtual, Cloud) 23
- 24. 24 THE BASICS DC/OS is … ● 100% open source (ASL2.0) + A big, diverse community ● An umbrella for ~30 OSS projects + Roadmap and designs + Docs and tutorials ● Not limited in any way ● Familiar, with more features + Networking, Security, CLI, UI, Service Discovery, Load Balancing, Packages,
- 25. SMACK Stack
- 26. © 2016 Mesosphere, Inc. All Rights Reserved. 26 SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications DC/OS Sensors Devices Clients
- 27. © 2016 Mesosphere, Inc. All Rights Reserved. 27 Apache Kafka ØMQ, RabbitMQ, Disque (Redis- based), etc. fluentd, Logstash, Flume Akka streams cloud-only: AWS SQS, Google Cloud Pub/Sub see also queues.io MESSAGE QUEUES
- 28. © 2016 Mesosphere, Inc. All Rights Reserved. 28 High-throughput, distributed, persistent publish-subscribe messaging system Originates from LinkedIn Typically used as buffer/de- coupling layer in online stream processing APACHE KAFKA
- 29. © 2016 Mesosphere, Inc. All Rights Reserved. 29 At most once—Messages may be lost but are never redelivered. At least once—Messages are never lost but may be redelivered. Exactly once—this is what people actually want, each message is delivered once and only once. DELIVERY GUARANTEES Murphy’s Law of Distributed Systems: Anything that can go wrong, will go wrong … partially!
- 30. © 2016 Mesosphere, Inc. All Rights Reserved. 30 SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications DC/OS Sensors Devices Clients
- 31. © 2016 Mesosphere, Inc. All Rights Reserved. 31 Apache Storm Apache Spark Apache Samza Apache Flink Apache Apex Concord cloud-only: AWS Kinesis, Google Cloud Dataflow STREAM PROCESSING
- 32. © 2016 Mesosphere, Inc. All Rights Reserved. 32 APACHE SPARK
- 33. © 2016 Mesosphere, Inc. All Rights Reserved. 33 Typical Use: distributed, large-scale data processing; micro-batching Why Spark Streaming? ● Micro-batching creates very low latency, which can be faster ● Well defined role means it fits in well with other pieces of the pipeline APACHE SPARK (STREAMING)
- 34. © 2016 Mesosphere, Inc. All Rights Reserved. 34 Micro-Batching EXECUTION MODEL Native Streaming
- 35. © 2016 Mesosphere, Inc. All Rights Reserved. 35 EXAMPLE: REAL-TIME TRACKING
- 36. © 2016 Mesosphere, Inc. All Rights Reserved. 36 GEO-ENABLED IoT
- 37. © 2016 Mesosphere, Inc. All Rights Reserved. 37 DATA FLOW
- 38. © 2016 Mesosphere, Inc. All Rights Reserved. 38 DEMO
- 39. © 2016 Mesosphere, Inc. All Rights Reserved. 39 SMACK stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications DC/OS Sensors Devices Clients
- 40. © 2017 Mesosphere, Inc. All Rights Reserved. 40 Keep it running!
- 41. © 2016 Mesosphere, Inc. All Rights Reserved. 41 SERVICE OPERATIONS ● Configuration Updates (ex: Scaling, re-configuration) ● Binary Upgrades ● Cluster Maintenance (ex: Backup, Restore, Restart) ● Monitor progress of operations ● Debug any runtime blockages
- 42. © 2017 Microsoft Corporation. All rights reserved. 本情報の内容（添付文書、リンク先などを含む）は、作成日時点でのものであり、予告なく変更される場合があります。 @dcos users@dcos.io /groups/8295652 /dcos /dcos/examples /dcos/demos chat.dcos.io @joerg_schad