Kubernetes + Docker + Elixir - Alexei Sholik, Andrew Dryga | Elixir Club Ukraine
1 like642 views
The document discusses issues related to Kubernetes pods failing to terminate for 22 days, indicating potential Docker bugs or resource limitations. It includes details about container readiness and liveness probes as well as diagnostic commands displaying system process states. Additional context about using 'tini' to handle zombie processes and runtime configurations is provided.
![2019-08-19T01:27:19.462500954Z [ERROR] Failed to restore
container ... with containerd: error getting containerd
task for container: OCI runtime state failed: runc did not
terminate sucessfully: runtime/cgo: pthread_create
failed: Resource temporarily unavailablenSIGABRT: abort](https://image.slidesharecdn.com/alcolightingtalk-191002135434/85/Kubernetes-Docker-Elixir-Alexei-Sholik-Andrew-Dryga-Elixir-Club-Ukraine-9-320.jpg)
![$ ps -ef | awk '{ print $8 $9 $10; }' | sort | uniq -c
...
4845 [epmd]<defunct>
4840 [erl_child_setup]<defunct>
3 [ext4-rsv-conver]
7200 [inet_gethost]<defunct>
...](https://image.slidesharecdn.com/alcolightingtalk-191002135434/85/Kubernetes-Docker-Elixir-Alexei-Sholik-Andrew-Dryga-Elixir-Club-Ukraine-13-320.jpg)
![PID 1
USER PID PPID STAT COMMAND COMMAND
default 1 0 S beam.smp {beam.smp} /opt/imaginator/releases/0.4.13/imaginator.sh -Bd -SDio 100 -- ...
default 137 1 Z bash [bash]
default 144 1 Z epmd [epmd]
default 145 1 S epmd /usr/local/lib/erlang/erts-10.4.4/bin/epmd -daemon
default 149 1 S erl_child_setup erl_child_setup 1048576
default 268 0 S bash bash
default 273 268 R ps ps -o user,pid,ppid,stat,comm,args](https://image.slidesharecdn.com/alcolightingtalk-191002135434/85/Kubernetes-Docker-Elixir-Alexei-Sholik-Andrew-Dryga-Elixir-Club-Ukraine-14-320.jpg)
![@@ -70,6 +70,10 @@ ENV REPLACE_OS_VARS=true
# Bash is required by Distillery
RUN apk add --update --no-cache bash
+ # Tini is used as the entrypoint to reap zombie processes.
+ # This is the same as using `docker run --init`.
+ RUN apk add --no-cache tini
+
@@ -85,6 +89,8 @@ WORKDIR /opt/${APPLICATION_NAME}
# Change user to "default" to limit runtime privileges
USER default
+ ENTRYPOINT ["/sbin/tini", "--"]](https://image.slidesharecdn.com/alcolightingtalk-191002135434/85/Kubernetes-Docker-Elixir-Alexei-Sholik-Andrew-Dryga-Elixir-Club-Ukraine-15-320.jpg)
Kubernetes + Docker + Elixir - Alexei Sholik, Andrew Dryga | Elixir Club Ukraine
- 2. KUBERNETES CAN'T TERMINATE POD FOR 22 DAYS
- 3. sudo systemctl restart docker did nothing
- 4. ktl delete pod ... --force --grace-period=0 kept containers running behind the scenes
- 5. docker stop COINTAINER_ID docker kill COINTAINER_ID did not stop containers either. (Docker bug?)
- 7. TOP
- 8. CPU UTILIZATION IS LOW TOO
- 9. 2019-08-19T01:27:19.462500954Z [ERROR] Failed to restore container ... with containerd: error getting containerd task for container: OCI runtime state failed: runc did not terminate sucessfully: runtime/cgo: pthread_create failed: Resource temporarily unavailablenSIGABRT: abort
- 11. # This probe is used once a while to check that container is ready to join the cluster readinessProbe: initialDelaySeconds: 40 periodSeconds: 15 timeoutSeconds: 30 failureThreshold: 6 exec: command: - /opt/talkinto_domain/bin/talkinto_domain - ping # This probe is used once a while to check that container is healthy livenessProbe: initialDelaySeconds: 30 periodSeconds: 30 timeoutSeconds: 30 failureThreshold: 2 exec: command: - /opt/talkinto_domain/bin/talkinto_domain - ping
- 13. $ ps -ef | awk '{ print $8 $9 $10; }' | sort | uniq -c ... 4845 [epmd]<defunct> 4840 [erl_child_setup]<defunct> 3 [ext4-rsv-conver] 7200 [inet_gethost]<defunct> ...
- 14. PID 1 USER PID PPID STAT COMMAND COMMAND default 1 0 S beam.smp {beam.smp} /opt/imaginator/releases/0.4.13/imaginator.sh -Bd -SDio 100 -- ... default 137 1 Z bash [bash] default 144 1 Z epmd [epmd] default 145 1 S epmd /usr/local/lib/erlang/erts-10.4.4/bin/epmd -daemon default 149 1 S erl_child_setup erl_child_setup 1048576 default 268 0 S bash bash default 273 268 R ps ps -o user,pid,ppid,stat,comm,args
- 15. @@ -70,6 +70,10 @@ ENV REPLACE_OS_VARS=true # Bash is required by Distillery RUN apk add --update --no-cache bash + # Tini is used as the entrypoint to reap zombie processes. + # This is the same as using `docker run --init`. + RUN apk add --no-cache tini + @@ -85,6 +89,8 @@ WORKDIR /opt/${APPLICATION_NAME} # Change user to "default" to limit runtime privileges USER default + ENTRYPOINT ["/sbin/tini", "--"]
- 16. USER PID PPID STAT COMMAND default 1 0 S /sbin/tini -- /bin/sh -c bin/${APPLICATION_NAME} foreground default 6 1 S {beam.smp} /opt/talkinto_domain/releases/0.8.173/talkinto_domain.sh -Bd -SDio 100 -- ... default 146 1 S /usr/local/lib/erlang/erts-10.4.4/bin/epmd -daemon default 150 6 S erl_child_setup 1048576 default 259 150 S inet_gethost 4 default 260 259 S inet_gethost 4 default 261 259 S inet_gethost 4 default 262 259 S inet_gethost 4 default 10802 0 R ps -o user,pid,ppid,stat,args