Doing Big Data for Real with Docker
6 likes1,949 views
The document outlines a presentation on utilizing Docker and Mesosphere DC/OS for big data container orchestration, led by Elizabeth Lingg, an engineering manager at Mesosphere. It covers the advantages of treating a cluster as a single computational resource, the specifics of managing and deploying distributed applications like Kafka and Cassandra on DC/OS, and the tools available for configuration and management. The session includes a demo and an opportunity for questions, emphasizing the capabilities of DC/OS in big data environments.
Doing Big Data for Real with Docker
- 1. DOING BIG DATA FOR REAL WITH DOCKER MESOSPHERE DCOS Elizabeth Lingg elizabeth@mesosphere.io
- 2. AGENDA 1. Intro 2. Mesosphere, Docker, and DCOS Overview 3. Big Data Container Orchestration using DCOS and Docker 4. Demo 5. Q & A
- 3. INTRO Engineering Manager @ Mesosphere M.S. Computer Science with a Specialization in Artificial Intelligence from Stanford B.S. Computer Science with a Minor in Math, B.S. Policy and Management from Carnegie Mellon Experience in AI, Big Data, and Systems Enjoys applying Distributed Systems to Manage and Reason Over Large Amounts of Data
- 4. MESOS Provides primitives to author datacenter-native apps. PRIMITIVES Resources (cpu, mem, disk, ports) Asset fetching Task state tracking API for the datacenter
- 5. STATUS QUO IS STATIC PARTITIONING AND USE OF VIRTUAL MACHINES
- 6. MESOS LET US TREAT A CLUSTER OF NODES...
- 7. AS ONE BIG COMPUTER
- 8. Not as individual machines Not as VMs
- 9. BUT AS COMPUTATIONAL RESOURCES LIKE CORES, MEMORY, DISKS, ETC.
- 12. MOST MODERN APPLICATIONS ARE A WEB OF CONTAINERS
- 13. A CONTAINER ORCHESTRATION PLATFORM
- 14. Containerization in Mesos, a brief history
- 15. MESOSPHERE DCOS Software to provide a complete OS: init, cron, apt-get, discovery, routing Beautiful web UI and CLI Support Ecosystem of DCOS Services Mesos Master and Mesos Workers Running in Docker Containers
- 16. DCOS UI
- 17. DCOS CLI $ dcos Command line utility for the Mesosphere Datacenter Operating System (DCOS). The Mesosphere DCOS is a distributed operating system built around Apache Mesos. This utility provides tools for easy management of a DCOS installation. Available DCOS commands: config Get and set DCOS CLI configuration properties help Display command line usage information marathon Deploy and manage applications on the DCOS node Manage DCOS nodes package Install and manage DCOS software packages service Manage DCOS services task Manage DCOS tasks
- 18. BIG DATA DISTRIBUTED APPLICATIONS ON DCOS Mesos Master and Mesos Workers Running in Docker Containers Distributed Applications Running in Containers on the Mesos Workers Container Orchestration done by Apache Mesos Resource Allocation and Scaling Managed by Apache Mesos
- 19. BIG DATA DISTRIBUTED APPLICATIONS ON DCOS Popular Distributed Apps easily deployed on a single DCOS Cluster Kafka, Cassandra, HDFS, Spark, and other Big Data Services Health checks and failure recovery are automated
- 20. APPLICATION NETWORKING Interact with the CLI or REST API's to interact with the services Mesos DNS Resolution Docker Networking mainly done through host mode networking, works seamlessly
- 21. DATA SECURITY Services storing secure data run on private worker nodes in the cluster Private nodes can only be accessed through VPN As needed, services choose what is exposed through a proxy running on a public node Distributed Application can authenticate with the Master using Framework Authentication (Kerberos Option)
- 22. EXAMPLE: SIMPLE DOCKER APP ON DCOS { "id": "/mesosphere/cd-demo-app", "instances": 1, "cpus": 1, "mem": 512, "container": { "type": "DOCKER", "docker": { "image": "mesosphere/cd-demo-app:$tag", "network": "BRIDGE", "portMappings": [ { "servicePort": 28080, "containerPort": 80, "hostPort": 0, "protocol": "tcp" }<
- 23. EXAMPLE: CASSANDRA DCOS SERVICE FEATURES Managed node configuration Health Monitoring Rest API DNS Names for nodes Multiple Rings in one cluster
- 24. INSTALL $ dcos package install cassandra CUSTOMIZABLE INSTALL OPTIONS { "cassandra": { "cluster-name": "dev", "resources": { "cpus": 3.0, "mem": 6144, "disk": 30720 } } } $ dcos package install cassandra --options=options.json
- 25. INSTALLING
- 26. HEALTHY
- 27. REST API GET /node/all GET /health/cluster/report POST /node/{node}/replace POST /cluster/repair/start POST /scale/nodes?nodeCount={count}
- 28. DEMO!
- 29. Q & A