The Challenges of Container Configuration
1 like914 views
The document discusses the complexities of container configuration, particularly focusing on immutability, runtime vs build time, and the importance of proper labeling and metadata. It highlights the challenges in enforcing immutability in containers, considerations for orchestrators, and the management of properties for containerized applications. Suggestions include using tmpfs for temporary file storage, embedding Dockerfiles in images, and leveraging higher-level programming tools for generating configuration data.
![(without introducing more risk)
Gareth Rushgrove
$ docker run --read-only -p 5000:5000 garethr/flaskapp
Traceback (most recent call last):
File "app.py", line 14, in <module>
handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1)
File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__
BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__
logging.FileHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__
StreamHandler.__init__(self, self._open())
File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open
stream = open(self.baseFilename, self.mode)
IOError: [Errno 30] Read-only file system: '/app/app.log'](https://image.slidesharecdn.com/containerconfiguration-160905084138/85/The-Challenges-of-Container-Configuration-23-320.jpg)
![(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]](https://image.slidesharecdn.com/containerconfiguration-160905084138/85/The-Challenges-of-Container-Configuration-34-320.jpg)
![(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]
Where did this base image come from?](https://image.slidesharecdn.com/containerconfiguration-160905084138/85/The-Challenges-of-Container-Configuration-35-320.jpg)
![(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]
What packages are installed? At what version?
Where are those packages from?](https://image.slidesharecdn.com/containerconfiguration-160905084138/85/The-Challenges-of-Container-Configuration-36-320.jpg)
![(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]
What version of flask is this?](https://image.slidesharecdn.com/containerconfiguration-160905084138/85/The-Challenges-of-Container-Configuration-37-320.jpg)
![(without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]
What was in this folder at build time?](https://image.slidesharecdn.com/containerconfiguration-160905084138/85/The-Challenges-of-Container-Configuration-38-320.jpg)
![(without introducing more risk)
Gareth Rushgrove
kubernetes_pod { 'sample-pod':
ensure => present,
metadata => {
namespace => 'default',
},
spec => {
containers => [{
name => 'container-name',
image => 'nginx',
}]
},
}](https://image.slidesharecdn.com/containerconfiguration-160905084138/85/The-Challenges-of-Container-Configuration-84-320.jpg)
The Challenges of Container Configuration
- 1. (without introducing more risk) The Challenges of Container Configuration Puppet Gareth Rushgrove New capabilities and associated problems
- 2. (without introducing more risk) Gareth Rushgrove @garethr
- 3. (without introducing more risk) Gareth Rushgrove
- 4. (without introducing more risk) Configuration What is it and why should I care
- 5. (without introducing more risk) Gareth Rushgrove
- 6. (without introducing more risk) synonyms: design, grouping, marshalling Gareth Rushgrove
- 7. (without introducing more risk) Marshalling your containers Gareth Rushgrove
- 8. (without introducing more risk) - Immutability and containers - Runtime vs build time - Who configures the orchestrator? Gareth Rushgrove
- 9. (without introducing more risk) Mainly Docker and Kubernetes examples, but should be generally applicable Gareth Rushgrove
- 10. (without introducing more risk) Everything is immutable now? Assumptions vs reality
- 11. (without introducing more risk) Gareth Rushgrove
- 12. (without introducing more risk) Gareth Rushgrove ker run -d ubuntu:16.04 /bin/sh c "while true; do echo hello world; sleep 1; done"
- 13. (without introducing more risk) Gareth Rushgrove $ docker exec a7a01beb14de touch /tmp/surprise
- 14. (without introducing more risk) Gareth Rushgrove $ docker diff a7a01beb14de C /tmp A /tmp/surprise
- 15. (without introducing more risk) Containers are not immutable by default Gareth Rushgrove
- 16. (without introducing more risk) Containers are not immutable by default Gareth Rushgrove
- 17. (without introducing more risk) Gareth Rushgrove $ docker run --read-only -d ubuntu:16.04 /bin/sh -c "while true; do echo hello world; sleep 1; done"
- 18. (without introducing more risk) Gareth Rushgrove $ docker exec 379150b2cf05 touch /tmp/surprise touch: cannot touch '/tmp/surprise': Read-only file syste
- 19. (without introducing more risk) Suggestion Enable read-only where possible Gareth Rushgrove
- 20. (without introducing more risk) Many applications won’t start with a read-only filesystem Gareth Rushgrove
- 21. (without introducing more risk) Gareth Rushgrove 1 import logging 2 from logging.handlers import RotatingFileHandler 3 4 from flask import Flask 5 6 app = Flask(__name__) 7 8 @app.route('/') 9 def home(): 10 app.logger.info('log request') 11 return "We'll never get to here" 12 13 if __name__ == '__main__': 14 handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount 15 handler.setLevel(logging.INFO) 16 app.logger.addHandler(handler) 17 app.run(debug=True, host='0.0.0.0')
- 22. (without introducing more risk) Gareth Rushgrove 1 import logging 2 from logging.handlers import RotatingFileHandler 3 4 from flask import Flask 5 6 app = Flask(__name__) 7 8 @app.route('/') 9 def home(): 10 app.logger.info('log request') 11 return "We'll never get to here" 12 13 if __name__ == '__main__': 14 handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount 15 handler.setLevel(logging.INFO) 16 app.logger.addHandler(handler) 17 app.run(debug=True, host='0.0.0.0')
- 23. (without introducing more risk) Gareth Rushgrove $ docker run --read-only -p 5000:5000 garethr/flaskapp Traceback (most recent call last): File "app.py", line 14, in <module> handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1) File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__ BaseRotatingHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__ logging.FileHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open stream = open(self.baseFilename, self.mode) IOError: [Errno 30] Read-only file system: '/app/app.log'
- 24. (without introducing more risk) tmpfs support added in Docker 1.10 Gareth Rushgrove
- 25. (without introducing more risk) Gareth Rushgrove $ docker run --read-only --tmpfs /tmp -d ubuntu:16.04 /bin/sh -c "while true; do echo hello world; sleep 1; done"
- 26. (without introducing more risk) Gareth Rushgrove $ docker exec 222331443d28 touch /tmp/surprise
- 27. (without introducing more risk) Gareth Rushgrove $ docker diff 222331443d28
- 28. (without introducing more risk) Gareth Rushgrove $ docker exec 222331443d28 ls /tmp surprise
- 29. (without introducing more risk) Suggestion Use tmpfs only where needed Gareth Rushgrove
- 30. (without introducing more risk) Remember Without technical controls you only have social guarantees of immutability Gareth Rushgrove
- 31. (without introducing more risk) Build vs Run And the relationship between them
- 32. (without introducing more risk) Given an image - What machine built this image? - Are all the licenses compatible? - Who supports this image? - Does this image contain malware? Gareth Rushgrove
- 33. (without introducing more risk) Given a running container - Who built it? - How was it built? - What software does it contain? - Is the software up-to-date? Gareth Rushgrove
- 34. (without introducing more risk) Gareth Rushgrove FROM ubuntu:16.04 RUN apt-get update && apt-get install -y python-pip python-dev build-essent apt-get clean && rm -rf /var/lib/apt/lists/* RUN pip install flask COPY . /app WORKDIR /app ENTRYPOINT ["python"] CMD ["app.py"]
- 35. (without introducing more risk) Gareth Rushgrove FROM ubuntu:16.04 RUN apt-get update && apt-get install -y python-pip python-dev build-essent apt-get clean && rm -rf /var/lib/apt/lists/* RUN pip install flask COPY . /app WORKDIR /app ENTRYPOINT ["python"] CMD ["app.py"] Where did this base image come from?
- 36. (without introducing more risk) Gareth Rushgrove FROM ubuntu:16.04 RUN apt-get update && apt-get install -y python-pip python-dev build-essent apt-get clean && rm -rf /var/lib/apt/lists/* RUN pip install flask COPY . /app WORKDIR /app ENTRYPOINT ["python"] CMD ["app.py"] What packages are installed? At what version? Where are those packages from?
- 37. (without introducing more risk) Gareth Rushgrove FROM ubuntu:16.04 RUN apt-get update && apt-get install -y python-pip python-dev build-essent apt-get clean && rm -rf /var/lib/apt/lists/* RUN pip install flask COPY . /app WORKDIR /app ENTRYPOINT ["python"] CMD ["app.py"] What version of flask is this?
- 38. (without introducing more risk) Gareth Rushgrove FROM ubuntu:16.04 RUN apt-get update && apt-get install -y python-pip python-dev build-essent apt-get clean && rm -rf /var/lib/apt/lists/* RUN pip install flask COPY . /app WORKDIR /app ENTRYPOINT ["python"] CMD ["app.py"] What was in this folder at build time?
- 39. (without introducing more risk) The importance of time Gareth Rushgrove
- 40. (without introducing more risk) How often are images rebuilt? Gareth Rushgrove
- 41. (without introducing more risk) Rebuilding only on code change ignores environmental factors Gareth Rushgrove
- 42. (without introducing more risk) Versioning and metadata Gareth Rushgrove
- 43. (without introducing more risk) Gareth Rushgrove LABEL vendor="ACME Incorporated" com.example.is-beta com.example.version="0.0.1-beta" com.example.release-date="2015-02-12"
- 44. (without introducing more risk) Gareth Rushgrove LABEL vendor="ACME Incorporated" com.example.is-beta com.example.version="0.0.1-beta" com.example.release-date="2015-02-12" What time? What timezone?
- 45. (without introducing more risk) Gareth Rushgrove $ docker inspect -f "{{json .Config.Labels }}" 4fa6e0f0c678 | jq { "vendor": "ACME Incorporated", "com.example.is-beta": "", "com.example.version": "0.0.1-beta", "com.example.release-date": "2015-02-12" }
- 46. (without introducing more risk) Suggestion Decide upon and enforce metadata standards Gareth Rushgrove
- 47. (without introducing more risk) Gareth Rushgrove LABEL com.example.git.repository="https://github.com/pupp com.example.git.sha="dc123cfb5ed4dca43a84be34a99d7c com.example.build.time="2016-04-24T15:43:05+00:00" com.example.build.builder=“jenkins1.example.com" com.example.docs="https://github.com/puppetlabs/doc ...
- 48. (without introducing more risk) Suggestion Embed Dockerfiles in images Gareth Rushgrove
- 49. (without introducing more risk) Gareth Rushgrove $ docker inspect -f "{{json .Config.Labels }}" garethr/alpine | jq { "net.morethanseven.dockerfile": "/Dockerfile", }
- 50. (without introducing more risk) Gareth Rushgrove $ docker run -i -t garethr/alpine cat /Dockerfile FROM alpine LABEL net.morethanseven.dockerfile="/Dockerfile" RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile /
- 51. (without introducing more risk) Suggestion Provide an API for your containers Gareth Rushgrove
- 52. (without introducing more risk) Gareth Rushgrove $ docker inspect -f "{{json .Config.Labels }}" garethr/alpine | jq { "com.example.api.packages": "apk info -vv" }
- 53. (without introducing more risk) Gareth Rushgrove $ docker run -i -t garethr/alpine apk info -vv musl-1.1.11-r2 - the musl c library (libc) implementation busybox-1.23.2-r0 - Size optimized toolbox of many common UNIX alpine-baselayout-2.3.2-r0 - Alpine base dir structure and init openrc-0.15.1-r3 - OpenRC manages the services, startup and shu alpine-conf-3.2.1-r6 - Alpine configuration management scripts
- 54. (without introducing more risk)DEMO
- 55. (without introducing more risk) Who configures the scheduler? Higher level configuration
- 56. (without introducing more risk) Schedulers/orchestrators abstract you from - Where individual containers run - Balancing due to new resources - Balancing due to failed resources Gareth Rushgrove
- 57. (without introducing more risk) This results in a constraints based system Gareth Rushgrove
- 58. (without introducing more risk) Which means those constraints need to be explicit and correct Gareth Rushgrove
- 59. (without introducing more risk) Gareth Rushgrove $ docker daemon --label com.example.environment="production" --label com.example.storage="ssd"
- 60. (without introducing more risk) Gareth Rushgrove $ docker run -d -P -e constraint:storage==ssd --name db mysql
- 61. (without introducing more risk) Gareth Rushgrove 1 template: 2 metadata: 3 labels: 4 app: guestbook 5 tier: frontend 6 spec: 7 containers: 8 - name: php-redis 9 image: gcr.io/google_samples/gb-frontend:v4 10 resources: 11 requests: 12 cpu: 100m 13 memory: 100Mi 14 env: 15 - name: GET_HOSTS_FROM 16 value: dns 17 # If your cluster config does not include a dns service, th 18 # instead access environment variables to find service host 19 # info, comment out the 'value: dns' line above, and uncomm
- 62. (without introducing more risk) How do you manage properties for all of your hosts? Gareth Rushgrove
- 63. (without introducing more risk) Gareth Rushgrove $ docker daemon --label com.example.environment="production" --label com.example.storage="ssd" Does this machine really have an SSD? What if someone swaps the drive?
- 64. (without introducing more risk) Suggestion Use properties of hosts as labels Gareth Rushgrove
- 65. (without introducing more risk) Gareth Rushgrove $ facter | head -n 20 architecture => x86_64 domain => local facterversion => 2.4.6 fqdn => Pro.local gid => staff hardwareisa => i386 hardwaremodel => x86_64 hostname => Pro id => garethr interfaces => lo0,gif0,stf0,en0,p2p0,awdl0,en1,en2,bridge0,vboxnet0,vboxnet1,v ipaddress => 192.168.0.5 ipaddress_en0 => 192.168.0.5
- 66. (without introducing more risk) Gareth Rushgrove $ facter -j os | jq { "os": { "name": "Darwin", "family": "Darwin", "release": { "major": "14", "minor": "5", "full": "14.5.0" } } }
- 67. (without introducing more risk) Gareth Rushgrove $ docker daemon --label net.example.os=`facter operatingsystem` --label net.example.virtual=`facter is_virtual` --label net.example.kernel=`facter kernelversion` ...
- 68. (without introducing more risk) Orchestrators also tend to introduce new higher-level primitives Gareth Rushgrove
- 69. (without introducing more risk) Docker Networks, Kubernetes Services, ReplicationControllers, Chronos Jobs Gareth Rushgrove
- 70. (without introducing more risk) Many with imperative interfaces Gareth Rushgrove
- 71. (without introducing more risk) Gareth Rushgrove $ kubectl get pod mypod -o yaml | sed 's/(image: myimage):.*$/1:v4/' | kubectl replace -f -
- 72. (without introducing more risk) Gareth Rushgrove $ docker network create bob c0a0f4538d259515813b771264688d37aaedb41098379a0d73ec0ca08 $ docker network create bob Error response from daemon: network with name bob already
- 73. And everything configured in YAML Gareth Rushgrove
- 74. Code plus data has advantages over data alone Gareth Rushgrove
- 75. The language to represent the data should be a simple, data-only format such as JSON or YAML, and programmatic modification of this data should be done in a real programming language Gareth Rushgrove Borg, Omega, and Kubernetes, ACM Queue,Volume 14, issue 1 http://queue.acm.org/detail.cfm?id=2898444 “
- 76. Avoid repetition Combine external inputs Correctness Abstractions Gareth Rushgrove - - - -
- 77. Suggestion Use a higher level programming tool for generating config data Gareth Rushgrove
- 78. (without introducing more risk) Gareth Rushgrove jsonnet.org
- 79. (without introducing more risk) Gareth Rushgrove $ cat test.jsonnet // Example jsonnet file { person1: { name: "Alice", welcome: "Hello " + self.name + "!", }, person2: self.person1 { name: "Bob" }, }
- 80. (without introducing more risk) Gareth Rushgrove $ jsonnet test.jsonnet | jq { "person1": { "name": "Alice", "welcome": "Hello Alice!" }, "person2": { "name": "Bob", "welcome": "Hello Bob!" } }
- 81. (without introducing more risk) Gareth Rushgrove $ jsonnet test.jsonnet | json2yaml --- person1: name: "Alice" welcome: "Hello Alice!" person2: name: "Bob" welcome: "Hello Bob!"
- 82. (without introducing more risk)DEMO
- 83. (without introducing more risk) Gareth Rushgrove garethr/kubernetes
- 84. (without introducing more risk) Gareth Rushgrove kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, }
- 85. (without introducing more risk) Gareth Rushgrove apiVersion: v1 kind: Pod metadata: namespace: default name: sample-pod spec: container: - image: nginx name: container-name
- 86. (without introducing more risk) Gareth Rushgrove controller_service_pair { 'redis-master': app => 'redis', role => 'master', tier => 'backend', port => 6379, }
- 87. (without introducing more risk) Gareth Rushgrove apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis tier: backend role: master spec: ports: # the port that this service should serve on - port: 6379 targetPort: 6379 selector: app: redis tier: backend role: master --- apiVersion: v1 kind: ReplicationController metadata: name: redis-master # these labels can be applied automatically # from the labels in the pod template if not set labels: app: redis role: master tier: backend spec: # this replicas value is default # modify it according to your case replicas: 1 # selector can be applied automatically # from the labels in the pod template if not set # selector: # app: guestbook # role: master # tier: backend
- 88. (without introducing more risk)DEMO
- 89. (without introducing more risk) Conclusions New technology means old problems
- 90. (without introducing more risk) The difference between how you think something works and how it actually works risks hard-to-debug production issues Gareth Rushgrove
- 91. (without introducing more risk) Containers introduce new and old configuration problems Gareth Rushgrove
- 92. (without introducing more risk) Configuration management is the discipline aimed at minimising those risks Gareth Rushgrove
- 93. (without introducing more risk) Start with principles Gareth Rushgrove
- 94. (without introducing more risk) - Identification - Control - Status accounting - Verification Gareth Rushgrove Military Handbook Configuration Management Guidance MIL-HDBK-61B
- 95. (without introducing more risk) Apply them to your container based infrastructure today Gareth Rushgrove
- 96. (without introducing more risk) Questions? And thanks for listening