The pain and gains running Docker in live @Pipedrive
10 likes1,552 views
The document summarizes the company Pipedrive's experience with moving to Docker for their infrastructure and applications. Some key points: - They initially had issues with slow build/deploy times using Codeship and Tutum, taking over 15 minutes. This caused reliability problems. - They created a new Docker infrastructure "v2" using Jenkins for automation and Docker Swarm for scheduling to improve speed and reliability. - Optimizing the Dockerfile and using AUFS instead of Devicemapper reduced deployment times from over 5 minutes to under 5 minutes. - Health checks, unique service IDs, load testing, and kernel tuning were used to ensure high availability of services under heavy load. - Regular
![Improved Docker builds
First iteration:
FROM node
ENV SERVICE_NAME=statistics
ENV SERVICE_DESC="Statistics"
ENV SERVICE_TAGS=statistics
ENV SERVICE_CHECK_HTTP=/health
ENV SERVICE_CHECK_INTERVAL=10s
ENV SERVICE_CHECK_TIMEOUT=5s
EXPOSE 8000
WORKDIR /src
COPY . /src/
RUN npm install
CMD ["node", "."]
Improved:
FROM node:6-alpine
ENV SERVICE_NAME=statistics
SERVICE_DESC="Statistics"
SERVICE_TAGS=statistics
SERVICE_CHECK_HTTP=/health-statistics
SERVICE_CHECK_INTERVAL=10s
SERVICE_CHECK_TIMEOUT=5s
EXPOSE 8000
WORKDIR /src
USER node
CMD ["node", "."]
COPY libraries/ /src/
COPY src/ /src/](https://image.slidesharecdn.com/dockerhelsinki17-170121125006/85/The-pain-and-gains-running-Docker-in-live-Pipedrive-12-320.jpg)
The pain and gains running Docker in live @Pipedrive
- 1. The pain and gains running Docker in live @Pipedrive Renno Reinurm 17.01.17
- 2. ● Pipedrive helps small businesses control the complex selling process ● Founded in 2010 ● 30,000 paying customers worldwide ● 200+ employees ● Offices in Tallinn and Tartu New York, NY
- 3. Pipedrive helps small businesses control the complex selling process
- 4. Why to use Docker? ● Growth pains with Chef ● New language + new tools = entry barrier ● You write recipes seldom enough and forget how it’s done ● But it runs fine in test!
- 5. Early docker platform started with evaluating running docker inside Vagrant box. Instead we started to use custom built docker-machine. Lately moved to Docker4Mac
- 6. First use case for containers Provision on demand test environments per branch. Was implemented only for test coverage-suite execution environment. Lot of custom hacks to make it work.
- 7. Docker infrastructure v1 The first Docker builds using Codeship Docker CI beta The first usage of Tutum (Docker Cloud) as orchestration service
- 8. Yeah we were using Docker, but CI processes with Codeship was slow, Docker build itself took ~15minutes Deployment in Docker Tutum cluster took another ~10minutes Sometimes it was so slow we wondered if it still works Stability issues - we experienced “data loss” and “service downtime”
- 9. The Birth of Docker Infrastructure v2.0 Requirements: Improve the speed of CI processes Improve the reliability of Docker Infrastructure
- 10. Docker Infrastructure v2.0 Jenkins for automating processes Docker image builds Container deployment Docker Swarm Container Scheduler Shipyard Troubleshooting
- 11. Pain 1 You shall not build/test/deploy Docker container over 5 minutes Based on: xkcd.com
- 12. Improved Docker builds First iteration: FROM node ENV SERVICE_NAME=statistics ENV SERVICE_DESC="Statistics" ENV SERVICE_TAGS=statistics ENV SERVICE_CHECK_HTTP=/health ENV SERVICE_CHECK_INTERVAL=10s ENV SERVICE_CHECK_TIMEOUT=5s EXPOSE 8000 WORKDIR /src COPY . /src/ RUN npm install CMD ["node", "."] Improved: FROM node:6-alpine ENV SERVICE_NAME=statistics SERVICE_DESC="Statistics" SERVICE_TAGS=statistics SERVICE_CHECK_HTTP=/health-statistics SERVICE_CHECK_INTERVAL=10s SERVICE_CHECK_TIMEOUT=5s EXPOSE 8000 WORKDIR /src USER node CMD ["node", "."] COPY libraries/ /src/ COPY src/ /src/
- 14. Deployment process optimizations NB! https://docs.docker.com/engine/userguide/storagedriver/selectadriver/ Replacement of Devicemapper to AUFS reduced deployment process time 10x. There are still improvements possible: ● Handle Linux signals ● Parallel rolling updates https://teespring.com/sigkill
- 15. Pain 2 Consumers shall connect only to healthy services
- 16. Beware the service discovery corruption ● Always enable health checks ● Use unique health checks or validate output SERVICE_CHECK_HTTP=/health vs SERVICE_CHECK_HTTP=/statistics-health
- 17. Pain 3 - Every day maintenance of Jenkins jobs
- 18. Pain 4 Container shall handle 10 000 connections and constant high load.
- 19. https://youtu.be/PivpCKEiQOQ We deployed Killer-Container to the cluster and rescheduled it every time then it managed to crash the Docker host
- 21. Issues ● Linux kernel 3.13 ● Fluentd logging agent ● Graylog logging driver ● Kernel sysctl parameters ● Swap usage ● PEBKAC ○ "net.ipv4.ip_forward" => 0 ● WARNING: No memory limit support ● WARNING: No swap limit support ● WARNING: No kernel memory limit support ● WARNING: No oom kill disable support ● WARNING: No cpu cfs quota support ● WARNING: No cpu cfs period support
- 22. Service risk mitigation ● Number of nodes in cluster ● Spreading policies ● Multiple instances ● Memory limitations ● Healing policies ○ Autorestart ○ Reschedule
- 23. Gains Evolution of applications generic enough to run in multiple regions, environments Delivery time from idea to live From 2 weeks to 1 day Servers vs Services those be managed asynchronously
- 24. Statistics ~ 70 inhouse built Dockerized services ~ 90 Docker images ~ 500 containers running 3200 container deploys since October
- 26. Remember - Every Day 1 new container borns to stay @Pipedrive 30 container deployments
- 27. Recommendations for going Live with Docker ● You still need to take care of OS ● Read Github issues ● Read from the source ● Keep it up to date ● (Performance) Test it
- 28. Thank you! Give me your feedback @rreinurm