One thing I learned today — 4 January 2018

When using Kubernetes it is common to find pods stuck in the Terminating state. Often this is because the Docker container on the worker is stuck, and one needs to get rid of that container to remove the pod.

Today I discovered that sometimes Terminating pods remain in that state even when the container was actually removed. This can be checked either by using docker ps on the worker or by checking the logs of the pod — if the API server responds that the container could not be found it is safe to assume that the container is gone and the pod can be removed forcefully:

kubectl delete pod --force --grace-period=0

Of course, forcefully deleting pods with containers that really are stuck is not a good idea, because then Kubernetes will no longer track those pods.