Docker在豆瓣的实践 刘天伟-20160709
2 likes676 views
This document discusses Docker and its use for the Douban App Engine (DAE). It covers: - The history of adopting Docker for DAE applications from 2014 to 2016. - How DAE uses Docker to build and deploy over 400 application images across different environments. - Techniques used to optimize the Docker build process and reduce image sizes. - Integrating Docker with the DAE monitoring, logging, and maintenance systems.
Docker在豆瓣的实践 刘天伟-20160709
- 3. Docker • DAE App • App docker image • DAE Python2.7, Python3, Golang • online/qaci/sandbox/prelease/dev • container • Docker
- 4. • 2014 Q4: DAE Docker App • 2015 Q1: App • 2015 Q2: App Docker • 2015 Q4: QACI/Prelease Docker • 2016.06 - : Mesos + Marathon
- 5. (1) • (400+ APP) • web, service(pidl/thrift) • DAE App cron, daemon, mq worker, remoteshell, app-admin • QACI container docker image • Prelease/Sandbox container docker image • Docker pidlproxy, dpark, scribe-sink, syslog-ng, thrift-router, dnasmasq ...
- 6. (2) • • 100 200-400 docker image build • container 370 • 400+ • 7500 + • Prelease/Sandbox 200+
- 7. Douban App Engine For Docker
- 8. • container web service container container • container • Image DAE • Dev QACI Prelease Sandbox Online
- 9. DAE • PaaS Python Golang 400+ App • Web, PIDL Service, Thrift Service, Cron, Daemon, MQ, Runscript, Admin, Dpark, QACI, Prelease, Sandbox... App MySQL, Beansdb, Memcahce, MQ, MFS, CDN, Redis... Proj A Proj B Proj C $ dae create $ dae serve $ dae test $ dae deploy App Auto Scale Load Balancer Node Balance Elastic Pool Prelease Sandbox Profile
- 10. DAE-Docker Cron MQ Worker DaemonDaemonDaemon DaemonMQ Worker M S S zookeeper Mesos Marathon+ MQ: Beanstalkd New! Gunicorn: web._admin websocket- runscript - remoteshell - dbshell online offline http(s) LVS Nginx(lb) Nginx Gateway douban service thrift dae_thrift_ router pidl pidlproxy- client pidlproxy- server Router Layer unix domain socket Node Gate Docker Container Gunicorn- sync/async - multi-instances - master-worker puppet:conf mfs:permdir volume monitor/sa agent bridge/host App Node DAE API (python/golang) Auto Scale Distributed Trace Orchestration waylifebeansdb/mysql/mc/mq/redis/graphite... Bridge/Deploy Logs Resource Layer dev Vagrant Container DAE-SDK Docker Compose Container App-Test Jenkins mysql redis mc scribe beansdb Prelease Sandbox Golang Gevent daes
- 11. DAE CI/CD Github/Code pull request merged review test + pylint prelease/sandbox Make Archive Image Build Image Push App Node1 App Node2 App NodeN docker pull Build Static Jenkins:QACI app.yaml chk hook trigger deploy Auto Stage0 Stage Deploy Deploy Rolling Update cfg update container restart selftest Image Cached Rollback No Yes update conf irc/slack/email notify end Regi stry
- 12. Docker
- 13. 1. : CentOS 6.5 vs Gentoo • Centos 6.5 (2014.9) • Docker ( Gentoo) • Docker 1.2 Ubuntu/Centos • CentOS6.5 Gentoo • CentOS 6.5, Linux 2.6 -> Docker Daemon crash, Kernal Panic • DAE / -> Gentoo binpkg CentOS RPM • CentOS 2.6 Python 2.6 -> Python2.7 hack link • Gentoo • Gentoo Linux kernel 3.18.10(overlayfs merged)
- 14. 2. Docker (1) • Docker 1.2.0 / 1.3.2 : -> • Docker 1.4.1: production ready • zombie container (container 1 docker daemon container docker run container name DAE container ) • zombie container -> kill docker run process -> rename linkgraphdb -> cleanup container files
- 15. 2. Docker (2) • Docker 1.5.0 DAE • : push/pull Docker Registry V1 • image pre-build pre-pull ( ) • Docker 1.7.0: Docker • : docker daemon docker daemon 7-8G container • + docker daemon
- 16. 2. Docker (3) • Docker 1.8.1 6 • 1 : container /etc/hosts net=host container /etc/hosts /etc/hosts container • container /etc/hosts container hosts /var/ lib/container/${cid}/hosts `mount --bind` container namespace net=host : • volume /var/run/nscd lookup cache • /var/lib/docker/${cid}/hosts inode container
- 17. 2. Docker (4) • Docker 1.8.1 6 • 2 : volume-from data container data container volume container app • data container ready • 3: docker registry v2 push image buffering disk, image partent image id • Docker 1.10.1 • : `Removal In Progress` container container name • docker rename • Docker 1.10.3 + docker registry v2 (2016.2 ) push/pull
- 18. 3. Docker Registry v2 • distribution: docker >1.10 push/pull • • docker.douban : build push • dev.docker.douban : • distribution api, DAE app Image • nginx upstream round robin push image `digest invalid: provided digest did not match uploaded content` mfs layer size • nginx ip_hash • distribution sentry load balancer distribution distribution distribution mossefs: Imageredis:cache
- 19. 4. Docker (1): dm vs overlay • DeviceMapper • 2014.09 ~ 2015.06 • • build image 400+ App 100 build 200-400 build • • 1. docker daemon : ` --storage-opt dm.blkdiscard=false --storage-opt dm.mountopt=nodiscard` --> kernel crash rm image pool • 2. /var/lib/docker/devicemapper/data `direct-lvm` -> image build 180s -> 120s
- 20. 4. Docker (2): dm vs overlay • ref: http://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/
- 21. 4. Docker (3): dm vs overlay • Overlay • 2015.06 - overlay2 • • 1. AUFS Linux upstream AUFS • 2. Overlay AUFS Linux 3.18 COW docker image • • /var/lib/docker/overlay lvm Ext4 • build 180s -> 120s -> 100s
- 22. 4. Docker (4): dm vs overlay • Overlay • inode ; build image , `too many links` --> docker 1.12 + kernel 4.0 + overlay2 • • 1. base image hard link kernal links • 2. linux ext4: `#define EXT4_LINK_MAX 65000` • docker overlay overlay container - low-id: image id - uppper: container - work: overlay - merged: container , mount-bind image overlay image - image - Linux 7 copy regular file hard link.
- 23. 5. DAE-Docker Image (1)
- 24. 5. DAE-Docker Image (2) • DAE-Python App • sys-req.txt: gentoo binpkg • pip-req.txt : python pip requirements sys-req.txt pip-req.txt
- 25. 6. Docker Build (1) • 400+ App 100 build image 200-400 build • • 1. App Image 4GB+, app 180s , `maintainers: xxx@douban.com` 40s • 2. Dockerfile layer, merge RUN: 180s -> 40s • 3. Dockerfile Docker container run + commit : 40s -> 8s
- 26. 6. Docker Build (2) • 95% layer diff layer diff docker build Gentoo Image portage + runtime portage emerge binpkg, python app run-build Dockerfile volume Docker run + volume + commit Image : 40s -> 8s Image : 4.0GB -> 1.7GB
- 27. 6. Docker Build (3) • • 1. gentoo_with_portage portage docker container • 2. dockerfile docker run + commit docker build docker run volume gentoo_with_portage container container portage emerge binpkg • • 1. Image • 2. gentoo_with_portage container • 3. Dockerfile • 4. volume-from container `mount --bind` container gentoo_with_portage volume list - /usr/portage/ - /etc/portage/ - /var/lib/layman/ - /var/cache/edb/ - /usr/libexec/gcc/ - /usr/share/include/ - /usr/x86_64-pc-linux-gnu/ - /lib64/modules/ - /lib64/firmware/ - /usr/lib64/gcc/x86_64-pc-linux-gnu/
- 28. 7. • : • • 1. puppet • 2. ossetup app image /var/dae/app/xxx/ • 3. volume container • - volume inode container
- 29. 8. moosefs fuse (1) • : DAE app permdir mfs( fuse ) mfs volume container mfs- mount ( mfs master ) container mfs `Transport endpoint is not connected` container • • 1. docker volume mfs inode 1 mfsmount mfs /proc/self/mountinfo container /etc/self/mountinfo
- 30. 8. moosefs fuse (2) • • 2. docker volume linux `mount --bind ` • 3. Linux shared subtrees mount namespace mount umount • • 1. /fuse mount point : `mount --bind /fuse /fuse` • 2. /fuse mount mode shared mount `mount --make-shared /fuse` • 3. mfs mount /fuse/mfs • 4. container `-v /fuse:/fuse:shared` , shared /fuse mount namespace container mount namespace mount/umount , mfsmount container
- 31. 9. container • : cadvisor( ) -> dae-monitor-agent • • 1. docker ps/inspect/exec container docker dameon docker daemon • 2. docker api • cgroups: cpu/mem/network/blkio • docker container • /var/lib/docker/container/${cid}/hostconfig.json • /var/run/docker/*/${cid}/state.json daemon-agent container-agent oom-agent monitor data upload redis mc graphite DAE Monitor Aggregator API Icinga2 sentry email - version/info - daemon process - image storage - dm cpu/mem - mem/cpu/blkio - network - image version - fuse mount dae-monitor-agent
- 32. 10. container log • DAE Python: • monkey patch: logger handler stdout/stderr -> ScribeHandler , container log- driver=none • , DAE-go • container syslog-driver + syslog-ng + scribe/mfs app-container --log-driver=syslog --log-opt syslog-address=tcp://127.0.0.1:9614 --log-opt tag=‘scribe@dae_applog/container’ --log-opt tag='/fuse/dfs/permdirs/daegrade/test.log' syslog-ng-container - 9614 syslog-ng - tag `scribe@` python scribe mod for syslog-ng, log scribe-buffer - tag `/fuse` log mfs mfs scribe-buffer scribe-sink
- 33. 11. maintainance • image container(exited/stopped/dead) • image docker rmi -v -v volume-from • docker daemon `-H 0.0.0.0:4243` tls • Docker Daemon daemontools supervisor docker daemon • docker daemon