Docker and Maestro for fun, development and profit
22 likes8,707 views
The document discusses the use of Docker and Maestro for container orchestration and deployment management. Maestro automates the deployment of Docker containers based on user-defined configurations, allowing for simplified management of multi-host environments. It highlights its capabilities in handling service dependencies and streamlining the development process through lightweight containers and automated CI outputs.
![docker:$ docker -d -H tcp://0.0.0.0:4243
!
client:$ cat << EOF | python import docker
from pprint import pprint as pp
pp(docker.client.Client(‘tcp://docker:4243')
.images('quay.io/signalfuse/maestro-base'))
EOF
!
!
[{u’Created': 1391202535,
u’Id': u’37de13d273eb9a02cd64…’,
u’Repository':
u'quay.io/signalfuse/maestro-base',
u'Size': 155663843,
u'Tag': u'0.1.6',
u'VirtualSize': 774767942}]](https://image.slidesharecdn.com/dockerandmaestroforfundevelopmentandprofit-140221111118-phpapp01/85/Docker-and-Maestro-for-fun-development-and-profit-7-320.jpg)
![#!/usr/bin/env python
!
# This is my cool container’s “init script”
!
import os
from maestro.guestutils import *
!
os.execl(‘java’, ‘java’,
‘-jar’, ‘my-app.jar’,
‘-DlistenPort={}’.format(get_port(‘service’)),
‘-DzkServers={}’.format(
get_node_list(‘zookeeper’, ports=[‘peer’])))](https://image.slidesharecdn.com/dockerandmaestroforfundevelopmentandprofit-140221111118-phpapp01/85/Docker-and-Maestro-for-fun-development-and-profit-21-320.jpg)
![# Kafka
kafka:
image: quay.io/signalfuse/kafka:0.8.0
requires: [ zookeeper ]
env:
ZOOKEEPER_BASE: /lspe/kafka
RETENTION_HOURS: 48
broker-1:
ship: vm
ports: {broker: 9092, jmx: “7199:17199”}
# Keep persistent data on the host.
volumes:
/var/lib/kafka: /data/kafka
env:
BROKER_ID: 0
More flexibility in port mappings, volume bindings, and
environment variables definition not shown here.](https://image.slidesharecdn.com/dockerandmaestroforfundevelopmentandprofit-140221111118-phpapp01/85/Docker-and-Maestro-for-fun-development-and-profit-29-320.jpg)
Docker and Maestro for fun, development and profit
- 1. Docker and Maestro For fun, development and profit
- 2. Maxime Petazzoni Software Engineer at SignalFuse, Inc. (also, Jérôme’s cousin) ! max@signalfuse.com
- 3. Real-time monitoring, instrumentation, observability and analytics Still in “stealth” mode Get updates at www.signalfuse.com
- 4. “Docker is awesome!” –You, some time in the last hour (hopefully).
- 5. A versatile foundation Service or application containment, security, software delivery, host and environment isolation, …and so much more.
- 6. Power at your fingertips Complete control through the remote API Available programmatic clients like docker-py
- 7. docker:$ docker -d -H tcp://0.0.0.0:4243 ! client:$ cat << EOF | python import docker from pprint import pprint as pp pp(docker.client.Client(‘tcp://docker:4243') .images('quay.io/signalfuse/maestro-base')) EOF ! ! [{u’Created': 1391202535, u’Id': u’37de13d273eb9a02cd64…’, u’Repository': u'quay.io/signalfuse/maestro-base', u'Size': 155663843, u'Tag': u'0.1.6', u'VirtualSize': 774767942}]
- 8. Docker’s Achilles: orchestration Single-host is alright with links, but multi-host just isn’t there.
- 9. How do I orchestrate the deployment and control of a full, multi-host, Docker-based environment?
- 10. (And more importantly:) How do I make this process one and the same for development, testing and production environments?
- 11. Enter: Maestro The totally not scalable, pet project that solved my use case. (and maybe yours)
- 12. Maestro is actually MaestroNG, a re-invention of Kimbro Staken’s Maestro (formerly, dockermix)
- 13. Takes in a definition of services, their dependencies , configuration and target host… ! …and automates the deployment (and control) of their corresponding containers on these hosts.
- 14. Classic use case: a pool of “dumb” workers on your favorite cloud/hosting provider that just run Docker. ! No need to (ma)ssh into anything, no need to pre-configure anything. ! Everything is remote controlled.
- 15. Other typical use case: running all the components of your stack in a single, local virtual machine. ! Useful for development, integration testing, etc.
- 16. Philosophy: lightweight application/service containers. ! Represent and control your software stack and its dependencies. ! Docker images are the output of your CI process (automation!). ! Start fast, fail faster. Not for heavyweight, complex container “VMs”.
- 17. Each service instance (container) defines where it runs and which ports it exposes, among other things. ! Like Docker links, Maestro works by injecting this information in the container’s environment about each container’s service’s dependencies.
- 18. Let’s say broker-1 of kafka depends on ZooKeeper. Its environment will contain: MAESTRO_ENVIRONMENT_NAME = lspe SERVICE_NAME = kafka CONTAINER_NAME = broker-1 CONTAINER_HOST_ADDRESS = 192.168.10.2 ! ZOOKEEPER_ZK_NODE_1_HOST = 192.168.10.2 ZOOKEEPER_ZK_NODE_1_CLIENT_PORT = 2181 ZOOKEEPER_ZK_NODE_1_PEER_PORT = 2888 ZOOKEEPER_ZK_NODE_1_LEADER_ELECTION_PORT = 3888 ! KAFKA_BROKER_1_HOST = 192.168.10.2 KAFKA_BROKER_1_BROKER_INTERNAL_PORT = 9042 KAFKA_BROKER_1_BROKER_PORT = 9042 KAFKA_BROKER_1_JMX_INTERNAL_PORT = 7199 KAFKA_BROKER_1_JMX_PORT = 17199
- 20. Using this information, you can configure your application at container start time. ! If you like Python, Maestro helps you by providing a set of guest helper functions in maestro.guestutils to easily extract and use this data.
- 21. #!/usr/bin/env python ! # This is my cool container’s “init script” ! import os from maestro.guestutils import * ! os.execl(‘java’, ‘java’, ‘-jar’, ‘my-app.jar’, ‘-DlistenPort={}’.format(get_port(‘service’)), ‘-DzkServers={}’.format( get_node_list(‘zookeeper’, ports=[‘peer’])))
- 22. Dependency order is respected on start; inverse order on stop. ! Can be overridden to stop individual services or containers.
- 23. MyApp Start order: 1. ZooKeeper 2. Kafka 3. MyApp Kafka ZK Stop order: 1. MyApp 2. Kafka 3. ZooKeeper Works on subsets of services too.
- 24. So how do you wield this power? A bit clunkily, with YAML (and a bit of Jinja2). ! ! ! (sorry)
- 25. # Yay, YAML! name: lspe ! registries: # Define custom image registries for # private registries, with credentials. ! ships: # Declare each target host. # (Docker daemon locations) ! services: # Declare each service, their # instances, dependencies and # configuration
- 26. registries: # Quay.io with Maestro robot account quay.io: registry: https://quay.io/v1/ email: maestro@signalfuse.com username: signalfuse+maestro password: {{ env.SUPER_SECRET }} When starting a container, Maestro will automatically login and pull the image from the right place if the image name matches a configured registry.
- 27. ships: # Local virtual machine vm: ip: 192.168.10.2 docker_port: 4243 timeout: 10 # Slow VM is slow # A shorter form… vm2: {ip: 192.168.10.3, timeout: 5} Ships carry containers and are referred to by name in the configuration.
- 28. services: # ZooKeeper zookeeper: image: quay.io/signalfuse/zookeeper:3.4.5 ! # Our zoo isn’t too wild, # only one keeper is enough. zk-node-1: ship: vm ports: client: 2181 peer: 2888/tcp leader_election: “3888/tcp:3888/tcp” # Keep persistent data on the host. volumes: /var/lib/zookeeper: /data/zookeeper # Environment can be passed-in too. env: JVM_FLAGS: “-Xmx1g”
- 29. # Kafka kafka: image: quay.io/signalfuse/kafka:0.8.0 requires: [ zookeeper ] env: ZOOKEEPER_BASE: /lspe/kafka RETENTION_HOURS: 48 broker-1: ship: vm ports: {broker: 9092, jmx: “7199:17199”} # Keep persistent data on the host. volumes: /var/lib/kafka: /data/kafka env: BROKER_ID: 0 More flexibility in port mappings, volume bindings, and environment variables definition not shown here.
- 30. See README.md for full syntax details and features https://github.com/signalfuse/maestro-ng/blob/master/README.md
- 31. Demo time! Be prepared for it to fail, because demos always do.
- 32. What’s next? More flexible service status detection (not only port pinging) Soft and hard service dependencies Parallel startup of independent services and instances of a service
- 33. That’s it! Thanks for listening! :) github.com/dotcloud/docker-py github.com/signalfuse/maestro-ng
- 34. SignalFuse is hiring world class engineers! jobs@signalfuse.com