Troubleshooting Tips from a Docker Support Engineer - Jeff Anderson, Docker

Troubleshooting
Jeff Anderson
Developer Support Engineer at
Docker
@programm3rq

Troubleshooting Basics
Common Issues
○ Volumes
○ Networking
○ TLS
Advanced Troubleshooting
Techniques
Troubleshooting

1. Characterization
2. Hypothesis
3. Test & Observe
Troubleshooting Basics

Volumes
Common Issues and
Questions

Minecraft Server
● Single Java Process
● Stores game world
state on disk
● Listens on port 25565
Enthusiast/Power
User/Tinkerer
Bob

FROM java:7
ADD minecraft_server.1.10.2.jar /
RUN mkdir -p /opt/minecraft
RUN echo "eula=true" > /opt/minecraft/eula.txt
EXPOSE 25565
WORKDIR /opt/minecraft
CMD java -jar /minecraft_server.1.10.2.jar
Minecraft Dockerfile

$ docker build -t mc:1.10.2 .
$ docker run -d --name old
-p 25565:25565
mc:1.10.2
Minecraft Build and Run

FROM java:7
ADD minecraft_server.1.11.2.jar /
RUN mkdir -p /opt/minecraft
RUN echo "eula=true" > /opt/minecraft/eula.txt
EXPOSE 25565
WORKDIR /opt/minecraft
CMD java -jar /minecraft_server.1.11.2.jar
Minecraft Dockerfile (updated)

$ docker build -t mc:1.11.2 .
$ docker stop old
$ docker run -d --name new
-p 25565:25565
mc:1.11.2
Minecraft Build and Run (updated)

Where did my stateful
minecraft data go?!
Bob

Storing important data
A volume is a directory on the host
that is made available to a container.
Docker does this with a bind mount.
Volumes

$ mount -o bind /opt/source /opt/destination
$ touch /opt/source/test
$ ls -li /opt/source/* /opt/destination/*
497080 -rw-r--r-- 1 root root 0 Apr 9 01:37 /opt/destination/test
497080 -rw-r--r-- 1 root root 0 Apr 9 01:37 /opt/source/test
$ ls -lid /opt/source/ /opt/destination/
500424 drwxr-xr-x 2 root root 4096 Apr 9 01:37 /opt/destination/
500424 drwxr-xr-x 2 root root 4096 Apr 9 01:37 /opt/source/
Bind Mount

Three Types
1. Host volume "I want my data to be here specifically"
2. Named Volume "I want to refer to my data later easily"
3. Anonymous Volume "I just want a volume"
Volumes

# Host Volume
$ docker run -v /opt/hostpath:/container/data …
# Named Volume
$ docker run -v important_stuff:/container/data …
# Anonymous Volume
$ docker run -v /container/data …
Volume Types

Bob
Minecraft data should
go in a volume.

$ docker diff old
…
C /opt/minecraft
A /opt/minecraft/server.properties
A /opt/minecraft/world
A /opt/minecraft/world/region
A /opt/minecraft/world/region/r.0.0.mca
…
Put data in a volume

$ docker volume create minecraft
$ docker create --name new
-p 25565:25565
-v minecraft:/opt/minecraft
mc:1.11.2
$ docker cp old:/opt/minecraft minecraft
$ docker cp minecraft new:/opt/
$ docker start new
Put data in a volume

Use volumes to
designate where
stateful data goes
Bob

Local dev environment
Ubuntu 16.04 desktop
Wants to use Docker in
her development
workflow
Ruby Developer
Jane

Useful for local development
Jane uses RubyMine
Wants code auto-reload with the rerun gem
Host Volumes

FROM ruby
RUN gem install sinatra sqlite3 rerun
COPY . /app/code
WORKDIR /app/code
EXPOSE 4567
CMD rerun 'ruby server.rb -o 0.0.0.0'
Ruby App Dockerfile

$ docker build -t my_sinatra_app .
$ docker run -p 4567:4567 --name webdev
-v /home/jane/code:/app/code my_sinatra_app
23:30:18 [rerun] Code launched
/usr/local/bundle/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:9
1:in `initialize': no such table: config
…
Jane's Ruby App

Useful for local development
This development environment needs a test database.
By default, it creates an sqlite3 file called test.db
This can be initialized with the 'init.sql' file in the project
Host Volumes

$ sqlite3 -bail test.db < init.sql
Error: near line 1: attempt to write a readonly database
Jane's Ruby App

Ruby Developer
JaneJane
File Permissions

Permission and ownership issues are dealt with in the
same way with and without docker.
The numeric uid is what matters.
Permissions and Ownership

$ sqlite3 -bail test.db < init.sql
Error: near line 1: attempt to write a readonly database
$ ls -lin
…
6721104 -rw-r--r-- 1 1000 1000 163 Apr 18 2017 init.sql
6721145 -rw-r--r-- 1 0 0 0 Apr 18 2017 test.db
Jane's Ruby App

Characterization and Hypothesis
● Files created by the container are owned by uid 0
● The image's default user is uid 0
● test.db file permissions are 0644
● sqlite3 is running as uid 1000 (jane)
Hypothesis: this is a normal permissions/ownership
issue.

Characterization and Hypothesis
Do these:
● chown 1000 test.db
● run container as uid 1000
Avoid these:
● chmod 777
● sudo sqlite3

examples of containerized process writing files
● database files
● pid files
● bytecode caching
● in-app file uploads
● plugin/theme installation
● log files

Docker for Mac
Docker for Mac shares files from macos host to hyperkit VM
This file sharing mechanism will ensure files written by
containers will always match your macos user id
Host Volumes

Ruby Developer
Volume Pro
JaneJane

Networking
Common Issues and
Questions

Working on a small
Python web application.
Early stages of
development.
Ready to Dockerize the
project.Web Developer
Small Company
Josh

from bottle import route, run, template
import socket
@route('/')
def index():
return str(socket.gethostname()) + 'n'
run(host='0.0.0.0', port=8000)
Application Code

FROM python:3-alpine
RUN pip install bottle
ADD . /code
WORKDIR /code
EXPOSE 8000
CMD ["python", "app.py"]
Application Dockerfile

$ docker build -t app .
$ docker run -d --name web -p 8000:8000 app
$ curl http://localhost:8000
d8939bc62a36
Running the python code

server {
listen 80;
server_name localhost;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
proxy_pass http://localhost:8000/;
}
}
nginx config file

FROM nginx:alpine
RUN rm -f /etc/nginx/conf.d/default.conf
ADD nginx.conf
/etc/nginx/conf.d/default.conf
nginx Dockerfile

$ docker build -t mynginx .
$ docker run -d --name nginx -p 80:80 mynginx
$ curl http://localhost/
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.11.10</center>
</body>
</html>
Running nginx

Web Developer
Small Company
Josh Unexpected 502 Error

502 - app.py and nginx
Networking
localhost eth0
nginx
localhost
eth0 -
172.18.0.6
nginx - 0.0.0.0:80
0.0.0.0:80
web
localhost
eth0 -
172.18.0.5
app.py - 0.0.0.0:8000
0.0.0.0:8000

502 Characterization and Hypothesis
● curl localhost:8000 does not work from nginx container
(connection refused)
● curl localhost:8000 works from the app container
● curl 172.18.0.5:8000 works from the nginx container
● curl 172.18.0.5:8000 works from the app container
Networking

Networking
localhost eth0
nginx
localhost
eth0 -
172.18.0.6
nginx - 0.0.0.0:80
0.0.0.0:80
web
localhost
eth0 -
172.18.0.5
app.py - 0.0.0.0:8000
0.0.0.0:8000
curlcurl

502 Characterization and Hypothesis
Hypothesis: nginx using the 'localhost' upstream is incorrect
Test: update the nginx config file with the container ip.
Networking

server {
listen 80;
location / {
proxy_pass http://172.18.0.5:8000/;
}
}
nginx config file

$ curl http://localhost
d8939bc62a36
Running the python code

Networking
localhost eth0
nginx
localhost
eth0 -
172.18.0.6
nginx - 0.0.0.0:80
0.0.0.0:80
web
localhost
eth0 -
172.18.0.5
app.py - 0.0.0.0:8000

Network Service Discovery
How will nginx discover the IP going forward?
Docker runs a resolver at 127.0.0.11.
It resolves container ips by their --name or --net-alias
Networking

server {
listen 80;
resolver 127.0.0.11;
location / {
proxy_pass http://web:8000/;
}
}
nginx config file updated

Web Developer
Container Networking
Specialist
Josh

TLS
Common Issues and
Questions

Docker EE
Docker Datacenter
Deploys internal apps
Devops Team at a big
company
Working on the Docker
Project
Steven

Universal Control Plane
● Implements the Docker Daemon API on port 443
● There is a web GUI as well
● You connect to it with a "client bundle"
TLS

$ ls
… ca.pem cert.pem key.pem … env.sh
$ cat env.sh
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH="$(pwd)"
export DOCKER_HOST=tcp://ucp.example.com:443
Client Bundle

$ source env.sh
$ docker run --rm -it alpine echo hello dockercon
hello dockercon
$ docker service create -p 80:80 nginx:alpine
ellhziigdmo2hae2z7wxuv4qt
Client Bundle

Installed New Certs
● Chrome no longer complains about the self signed
certificate
● docker run and docker service still work as they did
before
TLS

$ source env.sh
$ docker-compose up -d
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:590)
compose TLS issue

TLS issue reported after cert install
● TLS error when using compose
● Same endpoint works in browser
● Same endpoint works with `docker` CLI
Hypothesis: compose has different TLS client
expectations from this TLS endpoint
TLS

TLS issues don't need to be scary
Cheat sheet (check the following):
● Subject/Alt name match
● Full Chain of Trust
● Chain Root is trusted
TLS

● Subject/Alt name match correct
● Full Chain of Trust
TLS

openssl x509 -noout -text < 0.pem | grep 'Subject:|Issuer:'
Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
Subject: CN=ucp.example.com
cert Subject and Issuer

Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3

● Full Chain of Trust missing root
TLS

Subject: O=Digital Signature Trust Co., CN=DST Root CA X3

root:
Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
intermediary:
Subject: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
certificate:
Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
Subject: CN=ucp.example.com
cert chain

● Full Chain of Trust correct
TLS

● Full Chain of Trust correct
● Chain Root is trusted correct
TLS

$ source env.sh
$ docker-compose up -d
…
Creating network "acme_default" with the default driver
Creating acme_tomcat_1
Creating acme_apache_1
docker-compose working

TLS issue when using compose
● TLS works when using compose
● Same endpoint works in browser
● Same endpoint works with `docker` CLI
Python TLS client wants the certificate authority it trusts
to be a root certificate.
TLS

Advanced
Troubleshooting
Techniques

Amber keeps up pace by
being proactive
She has several general
troubleshooting tactics
that help characterize
issuesWorks at a big company
Has been a sysadmin,
developer, network admin
Currently technical lead on
the devops team
Amber

Tools - command line utilities
● socat - bidirectional communication over tcp, udp,
stdio, pipes, unix domain sockets, etc
● curl - make web requests
● jq - parse, filter, create json text
● regular network tools - iptables, ipvsadm, route, ip,
arp, tcpdump, ifconfig
● nsenter - enter a namespace
Amber's Toolbox

Tools - command line utilities
● Nico Kabar's netshoot container:
○ https://github.com/nicolaka/netshoot
○ docker pull nicolaka/netshoot
● Jérôme Petazzoni's nsenter
○ https://github.com/jpetazzo/nsenter
Amber's Toolbox

$ socat -v tcp4-listen:5566,bind=127.0.0.1,reuseaddr,fork
unix-connect:/var/run/docker.sock
$ docker -H 127.0.0.1:5566 ps
MITM docker socket traffic

$ socat -v tcp4-listen:5566,bind=127.0.0.1,reuseaddr,fork unix-connect:/var/run/docker.sock
> 2017/04/16 10:38:09.400245 length=131 from=115 to=245
GET /v1.26/containers/json HTTP/1.1r
Host: 127.0.0.1:5566r
User-Agent: Docker-Client/17.03.0-ce (darwin)r
Accept-Encoding: gzipr
r
< 2017/04/16 10:38:09.401486 length=197 from=199 to=395
HTTP/1.1 200 OKr
Api-Version: 1.26r
Content-Type: application/jsonr
Date: Sun, 16 Apr 2017 15:38:09 GMTr
Docker-Experimental: truer
Server: Docker/17.03.0-ce (linux)r
Transfer-Encoding: chunkedr
…
MITM docker socket traffic

$ curl -s --unix-socket /var/run/docker.sock
http::/containers/json | jq '.[].Names[0]'
"/focused_tesla"
"/exciting_einstein"
"/web"
"/app"
docker ps with curl | jq

$ PID=$(docker inspect --format {{.State.Pid}} happy_tesla)
$ nsenter -n -t $PID iptables -nL
$ nsenter -t `pidof dockerd` -m nsenter
--net=/var/run/docker/netns/ingress_sbox ipvsadm -l
$ for i in /var/run/docker/netns/* ; do nsenter -t
`pidof dockerd` -m nsenter --net=$i ifconfig; done
nsenter

Techniques
Host A container networking is working
Host B container networking is not
They are seemingly identical
How to identify the differences?
graphical diff!
Amber's Toolbox

Techniques - How to Ask a Question
Amber's Toolbox
<statement of observation>
|---------------------------|
| demonstration of relevant observations
|---------------------------|
<question>

Amber's Toolbox
<statement of observation>
|---------------------------|
| demonstration of relevant observations
|---------------------------|
<question>
Characterization
Hypothesis

Amber's Toolbox
I'm getting a 502 error when I hit the staging acmecorp endpoint
$ curl -vkL https://staging.internal.acmecorp.com/_ping/
…
Is there a deploy happening now?

Becoming a Troubleshooting Pro
● Docker Forums
https://forums.docker.com/
● Docker Community Slack
https://dockr.ly/community
What you can do

THANK YOU
Be a troubleshooting pro!
@docker #dockercon
Jeff Anderson @programm3rq

Troubleshooting Tips from a Docker Support Engineer - Jeff Anderson, Docker

More Related Content

What's hot (20)

Similar to Troubleshooting Tips from a Docker Support Engineer - Jeff Anderson, Docker (20)

More from Docker, Inc. (20)

Recently uploaded (20)

Troubleshooting Tips from a Docker Support Engineer - Jeff Anderson, Docker