Episode 3: Kubernetes and Big Data Services
Download as PPTX, PDF4 likes1,772 views
The document discusses the integration of Kubernetes with big data services, highlighting its use in building scalable solutions for data processing and analytics. It covers various components of the Kubernetes ecosystem, including automation, resource management, and the SMACK stack (Spark, Mesos, Akka, Cassandra, Kafka). Additionally, it provides links for resources, demonstrations, and community support related to Kubernetes and data services.
![© 2018 Mesosphere, Inc. All Rights Reserved.
Plans
dcos hdfs --name=hdfs plan status deploy
deploy (serial strategy) (COMPLETE)
├─ journal (serial strategy) (COMPLETE)
│ ├─ journal-0:[node] (COMPLETE)
│ ├─ journal-1:[node] (COMPLETE)
│ └─ journal-2:[node] (COMPLETE)
├─ name (serial strategy) (COMPLETE)
│ ├─ name-0:[node, zkfc] (COMPLETE)
│ └─ name-1:[node, zkfc] (COMPLETE)
└─ data (serial strategy) (COMPLETE)
├─ data-0:[node] (COMPLETE)
├─ data-1:[node] (COMPLETE)
└─ data-2:[node] (COMPLETE)](https://image.slidesharecdn.com/webinariii-kubernetesbigdata1-180425162742/85/Episode-3-Kubernetes-and-Big-Data-Services-25-320.jpg)
Episode 3: Kubernetes and Big Data Services
- 1. Kubernetes and Big Data Services @joerg_schad @gaunetes @dcos
- 2. Chris Gaun PMM at Mesosphere / Kubernetes Expert / CNCF Ambasador ● Previous to that Gartner analyst covering public IaaS ● Kubernetes community for 3 years
- 3. © 2018 Mesosphere, Inc. All Rights Reserved. Mesosphere DC/OS at KubeCon EU ● Mesosphere - Platinum Sponsor ● Many presentation: container storage, ML, HDFS ● Demoing smart city application 3
- 4. Jörg Schad Technical Community Lead / Developer ● Core Mesos developer at Mesosphere ● Passions are deep learning, distributed data systems, and data analytics
- 5. Bootcamp: Building Kubernetes-as-a-Service at Scale, Anywhere ● Episode 1: Building Kubernetes-as-a-Service at Scale ● Episode 2: Deploying Kubernetes at Scale with DC/OS ● Episode 3: Kubernetes and Big Data Services ● Episode 4: Operating Kubernetes at Scale with DC/OS ● End-to-end components and best practices ● Automated management of Kubernetes ● Connecting Kubernetes to Big Data services ● Delivering an entire Kubernetes solution
- 6. 6 Star / Clone Github 1. Go to Kubernetes DC/OS quickstart 2. Search “DC/OS Kubernetes Quickstart Github” or https://github.com/mesospher e/dcos-kubernetes-quickstart 3. Live demo https://github.com/dcos/demos /tree/master/flink-k8s/1.11
- 7. 7 Sign Up For Slack 1. Slack URL: https://chat.dcos.io/ 2. Join #kubernetes channel 3. OSS support / feedback
- 8. © 2017 Mesosphere, Inc. All Rights Reserved. 8 MapReduce is crunching Data Ancient Times...
- 9. © 2016 Mesosphere, Inc. All Rights Reserved. 9 But then business demanded FAST DATA We need to turn faster! Today...
- 10. © 2016 Mesosphere, Inc. All Rights Reserved. 10 Fast Data Batch Event ProcessingMicro-Batch Days Hours Minutes Seconds Microseconds Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations
- 11. © 2016 Mesosphere, Inc. All Rights Reserved. 11 The SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Apache Mesos/ DC/OS Sensors Devices Clients
- 13. © 2016 Mesosphere, Inc. All Rights Reserved. 13 The SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Flink ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Apache Mesos/ DC/OS Sensors Devices Clients
- 14. © 2017 Mesosphere, Inc. All Rights Reserved. 14 Challenges
- 15. © 2018 Mesosphere, Inc. All Rights Reserved. 15 Datacenter Typical Datacenter siloed, over-provisioned servers, low utilization Kubernetes Jenkins Kafka Spark Cassandra
- 16. 3 AM Typical Datacenter siloed, over-provisioned servers, low utilization Kubernetes Jenkins Kafka Spark Cassandra
- 17. DC1 DC2 DC3 3 AM
- 18. © 2017 Mesosphere, Inc. All Rights Reserved. 18
- 19. © 2018 Mesosphere, Inc. All Rights Reserved. 19 Datacenter Typical Datacenter siloed, over-provisioned servers, low utilization Mesos/ DC/OS automated schedulers, workload multiplexing onto the same machines Kubernetes Jenkins Kafka Spark Cassandra
- 20. • Brings “as-a-Service” automation to any application technology on any infrastructure • Organizations Run All Types of Container Management as-a- Service Using Mesos: "(Netflix) launches up to 500,000 containers and 200,000 clusters/day" -Netflix OSS, on using Titus container management ontop Mesos
- 21. © 2018 Mesosphere, Inc. All Rights Reserved. 21 DC/OS PHYSICAL INFRASTRUCTURE MICROSERVICES, CONTAINERS, & DEV TOOLS VIRTUAL MACHINES PUBLIC CLOUDS DATA SERVICES, MACHINE LEARNING, & AI Security & Compliance Application-Aware Automation Multitenancy Hybrid Cloud Management 100+ MORE DatacenterEdge Datacenter and Cloud as a Single Computing Resource Powered by Apache Mesos 20+ MORE
- 22. © 2018 Mesosphere, Inc. All Rights Reserved. 22
- 23. © 2018 Mesosphere, Inc. All Rights Reserved. Two-level Scheduling 1. Agents advertise resources to Master 2. Master offers resources to Framework 3. Framework rejects / uses resources 4. Agent reports task status to Master 23 MESOS ARCHITECTURE Mesos Master Mesos Master Mesos Master Mesos AgentMesos Agent Service Cassandra Executor Cassandra Task Kubernetes Scheduler Spark Executor Spark Task Mesos AgentMesos Agent Service Docker Executor Docker Task K8s Executor Kubelet Task Marathon Scheduler Kafka Scheduler
- 24. © 2018 Mesosphere, Inc. All Rights Reserved. Distributed Systems are ... HDFS Scheduler
- 25. © 2018 Mesosphere, Inc. All Rights Reserved. Plans dcos hdfs --name=hdfs plan status deploy deploy (serial strategy) (COMPLETE) ├─ journal (serial strategy) (COMPLETE) │ ├─ journal-0:[node] (COMPLETE) │ ├─ journal-1:[node] (COMPLETE) │ └─ journal-2:[node] (COMPLETE) ├─ name (serial strategy) (COMPLETE) │ ├─ name-0:[node, zkfc] (COMPLETE) │ └─ name-1:[node, zkfc] (COMPLETE) └─ data (serial strategy) (COMPLETE) ├─ data-0:[node] (COMPLETE) ├─ data-1:[node] (COMPLETE) └─ data-2:[node] (COMPLETE)
- 26. Server Server Server Server Server Server Server Server Server Server as-a-Service Installation KubernetesOne-Click 20+ MORE Cloud Native Services
- 27. ServerServer Server as-a-Service Automated Self Healing Server Server Server Server Server Server Server Server KubernetesZero Touch 20+ MORE Cloud Native Services
- 28. © 2018 Mesosphere, Inc. All Rights Reserved. 28 Why {Spark, HDFS, ..} on K8s today? Kelsey Hightower Kubernetes Thought Leader Ranked #1 K8s Influencer Staff Developer Advocate PM & Chief Advocate Today Big Data on K8s is more DIY Top Kubernetes Advocate
- 29. © 2018 Mesosphere, Inc. All Rights Reserved. 29 SMACK Stack Generator Display 1. Financial data created by generator 2. Written to Kafka topics 3. Kafka Topics consumed by Spark or Flink 4. Results written back into Kafka stream (another topic) 7. Results displayed
- 30. 30 Star / Clone Github 1. Go to Kubernetes DC/OS quickstart 2. Search “DC/OS Kubernetes Quickstart Github” or https://github.com/mesospher e/dcos-kubernetes-quickstart 3. Live demo https://github.com/dcos/demos /tree/master/flink-k8s/1.11
- 31. © 2018 Mesosphere, Inc. All Rights Reserved. 31 SMACK Stack Generator Display 1. Financial data created by generator 2. Written to Kafka topics 4. Results written back into Kafka stream (another topic) 5. Results displayed 3. Kafka Topics consumed and analyzed by Flink Kubernetes Cluster (running on top of DC/OS)
- 33. © 2018 Mesosphere, Inc. All Rights Reserved. 33 THANK YOU! ANY QUESTION S? @dcos users@dcos.io /groups/8295652 /dcos /dcos/examples /dcos/demos chat.dcos.io https://github.com/mesosphere/dcos-kubernetes-quickstart https://mesosphere.com/blog/another-kubernetes-service/