Postgre sql linuxcontainers by Jignesh Shah
1 like379 views
This document discusses Linux containers and PostgreSQL in Docker containers. It begins with an overview of containers, their advantages and disadvantages compared to virtual machines. It then discusses different implementations of containers like LXC and systemd-nspawn. A large portion of the document is dedicated to Docker containers - how to install, use images and volumes, and common commands. It concludes with best practices for running PostgreSQL in Docker containers, including mounting data volumes, linking containers, checking stats and processes.
![Dockerfile â Custom Recipe
⢠Container images are created using âDockerfileâ
⢠Build an image using the recipe file
⢠Push to a public or private registry (hub account login or private registry needed)
FROM centos:centos7.0.1406
RUN yum install -y epel-release
RUN yum install -y nginx
EXPOSE 80
CMD ["bash", "-l", "-c", "/usr/sbin/nginx -g "daemon off;""]
docker build ât jkshah/nginx ât .
docker push jkshah/nginx:latest](https://image.slidesharecdn.com/postgresqllinuxcontainersmeetup2015jan05-160106144426/85/Postgre-sql-linuxcontainers-by-Jignesh-Shah-18-320.jpg)
![Docker Images
⢠Docker Images are layered templates used by containers instances
⢠Container instance images are layered Copy on Write Images based on Docker
images
⢠You can mutate your local container images (unless you use --read-only flag)
⢠Not good for fast and frequent changing data
⢠Can select a different underlying type using --storage-driver
docker push jkshah/nginx:latest
Do you really want to push to public registry? [y/n]: y
The push refers to a repository [docker.io/jkshah/nginx] (len: 1)
d498b5680966: Pushed
4b1d16518ce1: Pushed
3185bef36db4: Pushed
2f7013eef4b8: Pushed
539eca37bade: Pushed
f1b10cd84249: Pushed
latest: digest:
sha256:46208d1f0393946b33c2bdce498168de507b7186a897f332bab0cffc1ea601c7 size:
10608](https://image.slidesharecdn.com/postgresqllinuxcontainersmeetup2015jan05-160106144426/85/Postgre-sql-linuxcontainers-by-Jignesh-Shah-19-320.jpg)
Postgre sql linuxcontainers by Jignesh Shah
- 1. PostgreSQL and Linux Containers Jignesh Shah Founding Team @appOrbit SF Bay Area PostgreSQL User Group â Jan 5th 2015
- 2. Disclaimer The views expressed in this presentation are my own and do not necessarily reflect the views of appOrbit.
- 3. About Jignesh (@jkshah) ⢠appOrbit ⢠My focus is on managing Application Data running in Containers ⢠VMware ⢠Lead and manage Postgres and Data Management teams at VMware for various products embedding PostgreSQL running in virtualized embedded instances ⢠Sun Microsystems ⢠Team Member of first published SpecJAppServer 2004 benchmark with PostgreSQL ⢠Performance of PostgreSQL on Solaris/Sun Servers ⢠Working with PostgreSQL community since 2005 ⢠http://jkshah.blogspot.com/2005/04/profiling-postgresql-using-dtrace-on_22.html ⢠Working with Container technologies (Solaris Zones) since 2004 ⢠http://jkshah.blogspot.com/2004/08/db2-working-under-solaris-10-zones_30.html
- 4. Agenda ⢠Containers ⢠Definition ⢠Early examples ⢠Linux Containers ⢠Underlying Technologies ⢠LXC ⢠Systemd-nspawn ⢠Docker Containers ⢠Installation ⢠Images ⢠Volumes ⢠PostgreSQL in Docker Container ⢠What it means ⢠Best practices
- 5. What are Containers? ⢠OS Level virtualization where kernel allows for multiple isolated user-space instances Operating System Bare Metal Server OS Bare Metal Server Hypervisor OS Operating System Bare Metal Server C C C C C OS Bare Metal Server Hypervisor OS C C C C
- 6. Advantages of Containers ⢠Lower footprint ⢠Very Quick Startup and Shutdown time ⢠Density ⢠Nesting
- 7. Disadvantages of Containers ⢠Same Kernel version ⢠Cannot run other OS natively ⢠Security (to be improved)
- 8. Where to use container? ⢠Recreate identical environment (cookie-cutter) ⢠Resource Grouping of specific processes in heavily loaded server ⢠Handling multiple versions of software applications ⢠Ephemeral application instances (Dev/Test) ⢠Many more
- 9. Implementations of Containers ⢠Chroot circa 1982 ⢠FreeBSD Jails circa 2000 ⢠Solaris Zones circa 2004 ⢠Meiosys â MetaClusters with Checkpoint/Restore 2004-05 ⢠Linux OpenVZ circa 2005 (not in mainstream Linux) ⢠AIX WPARs circa 2007 ⢠LXC circa 2008 ⢠Systemd-nspawn circa 2010-2013 ⢠Docker circa 2013
- 10. What makes containers possible? ⢠Process Group Isolation ⢠Filesystem Isolation ⢠Network Isolation ⢠CPU Isolation ⢠Memory Isolation ⢠Example â Solaris Containers ⢠Base implementation provided Process, Filesystem and Network Isolation ⢠Resource pools consisting of CPU, memory was originally used ⢠Branded Zones â Userland Library Isolation
- 11. What makes Linux containers possible ? ⢠cgroups ⢠Allows limitation and prioritization of resources (CPU, memory, block I/O, network, etc.) ⢠Namespace isolation ⢠Mount namespace ⢠PID namespace ⢠Network namespace ⢠UTS (Allows changing hostname, domainname) ⢠IPC Namespace ⢠User namespace ⢠LXC ⢠Combines kernelâs cgroup and namespaces to provide an isolated environment
- 12. LXC ⢠CentOS 7 + EPEL Repository set ⢠Commands available ⢠Quick Guide to use an LXC based container of busybox lxc-attach lxc-clone lxc-destroy lxc-ls lxc-stop lxc-usernsexec lxc-autostart lxc-config lxc-execute lxc-monitor lxc-top lxc-wait lxc-cgroup lxc-console lxc-freeze lxc-snapshot lxc-unfreeze lxc-checkconfig lxc-create lxc-info lxc-start lxc-unshare wget https://www.busybox.net/downloads/binaries/busybox-x86_64 -O busybox chmod a+x busybox PATH=$(pwd):$PATH lxc-create -t busybox -n mycontainer lxc-start -d -n mycontainer lxc-console ân mycontainer # (Use CTRL-A Q to exit console mode) lxc-stop -n mycontainer lxc-destroy -n mycontainer yum install epel-release yum install bridge-utils libvirt lxc lxc-templates
- 13. Systemd-nspawn ⢠Systemd ⢠Replacement of SysV init scripts ⢠Systemd-nspawn ⢠Used to run a command or OS in light weight namespace container ⢠Installed on most newer distros by default ⢠Commands available systemd-analyze systemd-delta systemd-nspawn systemd-ask-password systemd-detect-virt systemd-run systemd-cat systemd-cgls systemd-loginctl systemd-sysv-convert systemd-cgtop systemd-machine-id-setup systemd-coredumpctl systemd-notify systemd-tty-ask-password-agent systemd-inhibit systemd-stdio-bridge systemd-tmpfiles systemctl machinectl hostnamectl journalctl yum install systemd
- 14. Systemd-nspawn ⢠Quick guide to a container deployment using systemd-nspawn ⢠Useful tools machinectl status mycontainer systemd-cgls systemd-cgtop # Create an Image yum -y --releasever=7 --nogpg --installroot=/mycontainers/centos7 install systemd passwd yum fedora-release vim-minimal # Change the root password in the image systemd-nspawn -D /mycontainers/centos7 passwd exit # Start the container as if booting into the container image systemd-nspawn -D /mycontainers/centos7 âM mycontainer âb # Get into the container nsenter -m -u -i -n -p ât $PID machinectl login mycontainer
- 15. Trends of Container Technologies
- 16. Disruption of trends by Docker
- 17. Docker ⢠Installation ⢠Commands available with docker binary ⢠Quick Guide to use a docker based container attach build commit cp create diff events exec export history images import info inspect kill load login logout logs pause port ps pull push rename restart rm rmi run save search start stats stop tag top unpause version wait docker run --name mycontainer -e POSTGRES_PASSWORD=mysecretpassword -d postgres docker exec -ti mycontainer psql -U postgres docker stop mycontainer docker rm mycontainer docker rmi postgres yum install docker systemctl start docker
- 18. Dockerfile â Custom Recipe ⢠Container images are created using âDockerfileâ ⢠Build an image using the recipe file ⢠Push to a public or private registry (hub account login or private registry needed) FROM centos:centos7.0.1406 RUN yum install -y epel-release RUN yum install -y nginx EXPOSE 80 CMD ["bash", "-l", "-c", "/usr/sbin/nginx -g "daemon off;""] docker build ât jkshah/nginx ât . docker push jkshah/nginx:latest
- 19. Docker Images ⢠Docker Images are layered templates used by containers instances ⢠Container instance images are layered Copy on Write Images based on Docker images ⢠You can mutate your local container images (unless you use --read-only flag) ⢠Not good for fast and frequent changing data ⢠Can select a different underlying type using --storage-driver docker push jkshah/nginx:latest Do you really want to push to public registry? [y/n]: y The push refers to a repository [docker.io/jkshah/nginx] (len: 1) d498b5680966: Pushed 4b1d16518ce1: Pushed 3185bef36db4: Pushed 2f7013eef4b8: Pushed 539eca37bade: Pushed f1b10cd84249: Pushed latest: digest: sha256:46208d1f0393946b33c2bdce498168de507b7186a897f332bab0cffc1ea601c7 size: 10608
- 20. Docker Volumes ⢠Persists beyond the life of a Docker container ⢠VOLUME command in Dockerfile or ⢠Using âv using docker run command ⢠Automatically created if not already present during docker run ⢠Not part of docker push/pull operations ⢠Can select a non-local directory using --volume-driver ⢠Third party components required to get multi-host support (NFS, etc ) ⢠On CentOS with SELinux enabled need to set security context ⢠Different options using âv ⢠-v /hostsrc/data:/opt/data:ro # for read only volumes (default rw) ⢠-v /hostsrc/data:/opt/data:Z # Z â private volume, z â shared volume ⢠-v /etc/nginx.conf:/etc/nginx.conf # for mounting a single file only ⢠Volumes can be shared from another container using --volumes-from on same host ⢠Docker 1.9 gives first class status to Docker Volumes chcon -Rt svirt_sandbox_file_t /hostpath/pgdata
- 21. PostgreSQL in Docker container ⢠Quick Deployment: ⢠Check Deployment: ⢠Use inspect command to get more information about the container ⢠To check PostgreSQL system logs docker run --name mycontainer -v /hostpath/pgdata:/var/lib/postgresql/data -e POSTGRES_PASSWORD=mysecretpassword -d postgres docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8d720b31c1fe postgres "/docker-entrypoint.s" 5 minutes ago Up 5 minutes 5432/tcp mycontainer docker logs mycontainer The files belonging to this database system will be owned by user "postgres". This user must also own the server process. ⌠LOG: database system was shut down at 2016-01-04 22:58:40 UTC LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started docker inspect mycontainer
- 22. PostgreSQL in docker container ⢠Part of Postgres Dockerfile has ⢠Impacts ⢠Port is exposed only to other containers directly linking with this container ⢠Data is persistent only for the life of container (docker rm) but not easily accessible outside the container ⢠-v enables to access the data outside container ⢠Linking to PostgreSQL Server container ⢠Ports are exposed externally using âp option ⢠Verify using docker run --name myapp --link mycontainer:ds2db -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=mysecretpassword -p 8080:80 -d jkshah/dvdstore2 EXPOSE 5432 VOLUME /var/lib/postgresql/data docker port myapp 80/tcp -> 0.0.0.0:8080
- 23. PostgreSQL in docker container ⢠Check for statistics using ⢠Check for top processes in a container docker stats mycontainer myapp CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O myapp 0.01% 51.51 MB/1.924 GB 2.68% 54.42 kB/7.576 MB mycontainer 0.01% 133.3 MB/1.924 GB 6.93% 7.577 MB/54.42 kB docker top mycontainer UID PID PPID C STIME TTY TIME CMD polkitd 25313 10698 0 16:58 ? 00:00:00 postgres polkitd 25385 25313 0 16:58 ? 00:00:00 postgres: checkpointer process polkitd 25386 25313 0 16:58 ? 00:00:00 postgres: writer process polkitd 25387 25313 0 16:58 ? 00:00:00 postgres: wal writer process polkitd 25388 25313 0 16:58 ? 00:00:00 postgres: autovacuum launcher process polkitd 25389 25313 0 16:58 ? 00:00:00 postgres: stats collector process docker top myapp UID PID PPID C STIME TTY TIME CMD root 30747 10698 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30796 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30797 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30798 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30799 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND 48 30800 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND
- 24. PostgreSQL in docker container ⢠Alternate way to see Process Tree using systemd-cgls tool systemd-cgls ââ1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 ââuser.slice â ââuser-0.slice â ââsession-34.scope â ââ25129 sshd: root@pts/0 â ââ25131 -bash â ââ31397 systemd-cgls ââsystem.slice ââdocker-9457652d7f6ec24ebd95305e788fe39030b049deb22f240ee2b7383488d0c215.scope â ââ30747 httpd -D FOREGROUND â ââ30796 httpd -D FOREGROUND â ââ30797 httpd -D FOREGROUND â ââ30798 httpd -D FOREGROUND â ââ30799 httpd -D FOREGROUND â ââ30800 httpd -D FOREGROUND ââdocker-8d720b31c1fe0de77d3cd89942c1a72902b67d466f821ce63ee1271561a36451.scope â ââ25313 postgres â ââ25385 postgres: checkpointer process â ââ25386 postgres: writer process â ââ25387 postgres: wal writer process â ââ25388 postgres: autovacuum launcher process â ââ25389 postgres: stats collector process ââdocker.service â ââ10698 /usr/bin/docker daemon --selinux-enabled â ââ30739 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -contain...
- 25. PostgreSQL in Docker container ⢠Alternate way to see top containers using systemd-cgtop systemd-cgtop Path Tasks %CPU Memory Input/s Output/s / 100 0.9 1.5G - - /user.slice 3 0.9 70.7M - - /system.slice - 0.0 268.5M - - /system.slice/docker.service 2 0.0 31.9M - - /system.slice/nimbus.service 5 0.0 20.0K - - /system.slice/tuned.service 1 0.0 36.0K - - /system.slice/dock...49deb22f240ee2b7383488d0c215.scope 6 0.0 49.1M - - /system.slice/auditd.service 1 - 11.9M - - /system.slice/avahi-daemon.service 2 - 12.0K - - /system.slice/crond.service 1 - 92.0K - - /system.slice/dbus.service 1 - - - - /system.slice/dock...7d466f821ce63ee1271561a36451.scope 6 - 127.3M - - /system.slice/gssproxy.service 1 - - - - /system.slice/libvirtd.service 3 - - - - /system.slice/lvm2-lvmetad.service 1 - - - - /system.slice/polkit.service 1 - 16.0K - - /system.slice/postfix.service 3 - 3.5M - - /system.slice/rsyslog.service 1 - 2.1M - - /system.slice/sshd.service 1 - 3.0M - - /system.slice/system-getty.slice/getty@tty1.service 1 - - - - /system.slice/systemd-journald.service 1 - 16.0M - - /system.slice/systemd-logind.service 1 - 20.0K - - /system.slice/systemd-udevd.service 1 - 520.0K - - /system.slice/xe-linux-distribution.service 2 - 1.0M - - /user.slice/user-0.slice/session-34.scope 3 - - - -
- 26. PostgreSQL as a database server container ⢠Maybe you want a database server standalone ⢠Not all database clients will be on docker containers in the same host ⢠Need to limit memory usage ⢠Need different layout of how files are distributed (separage XLOG ) ⢠Use the âp option to make the port available even to non containers clients ⢠Use âm to limit memory usage by the DB server (by default it can see and use all) ⢠Note this does not set shared buffers automatically with the library image docker run --name mycontainer -m 4g -e POSTGRES_PASSWORD=mysecretpassword -v /hostpath/pgdata:/var/lib/postgresql/data -p 5432:5432 -d postgres
- 27. PostgreSQL in an enterprise environment ⢠However for a real production use case we would need ⢠Bigger shared memory configurations ⢠Need different layout of how files are distributed (separage XLOG ) ⢠Ability to backup the database ⢠Ability to setup replication ⢠etc ⢠In short we need a more custom image of PostgreSQL
- 28. Best Practices for custom image ⢠For production install customize the docker image ⢠Allocate proper memory limits - example 8GB ⢠All pagecache usage shows up as docker container memory usage ⢠Bump up shared buffers and other parameters as required ⢠Hint: use PostgreSQL 9.3 or later otherwise have to privileged containers ⢠http://jkshah.blogspot.com/2015/09/is-it-privilege-to-run-container-in.html ⢠Support multiple volumes in your image ⢠Pg_xlog ⢠PITR archives ⢠Tablespaces as required ⢠Full Backup directory ⢠PostgreSQL Extensions ⢠Setup replication support ⢠Out of box replication setup ⢠Monitoring Tool ⢠Your favorite monitoring agent
- 29. References ⢠http://www.freedesktop.org/wiki/Software/systemd/ ⢠https://linuxcontainers.org/ ⢠http://www.haifux.org/lectures/299/netLec7.pdf ⢠http://haifux.org/lectures/320/netLec8_final.pdf
- 30. Revolutionary platform to encapsulate and manage both legacy and new micro-services based applications with data and configurations running on containers and virtual machines in a private, public or hybrid cloud. Put your DevOps on hyper-drive! Efficiency and Velocity for Apps & Infrastructure We are HIRING !!!