{"id":77595,"date":"2026-02-10T17:38:22","date_gmt":"2026-02-10T12:08:22","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=77595"},"modified":"2026-02-24T18:10:18","modified_gmt":"2026-02-24T12:40:18","slug":"containers-lie-a-deep-dive-into-docker-shim-and-a-real-on-call-fix","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/containers-lie-a-deep-dive-into-docker-shim-and-a-real-on-call-fix\/","title":{"rendered":"Containers Lie | A Deep Dive into Docker-Shim and a Real On-Call Fix"},"content":{"rendered":"

In this BLOG I will share an incident that taught me how containers really work under the hood.<\/p>\n

Production Down –<\/h1>\n

Once I received production website down alert for one of my customer.
\nAs I checked the website was giving 502<\/p>\n

\"website-down\"

website-down<\/p><\/div>\n

Initial Checks –<\/h1>\n

I immediately logged in to the production host to investigate.<\/p>\n

The first thing I checked was the container that was running the production WordPress website.<\/p>\n

$ docker ps<\/em>
\nCONTAINER ID\u00a0 \u00a0 \u00a0 \u00a0 \u00a0IMAGE \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0COMMAND \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0CREATED \u00a0 \u00a0 \u00a0 \u00a0 \u00a0STATUS \u00a0 \u00a0 \u00a0 \u00a0 \u00a0PORTS\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 NAMES<\/em>
\n7f3a9c1b2d91 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0wordpress:6.4-apache \u00a0 \u00a0 \u00a0 \u00a0 \u00a0“docker-entrypoint.s\u2026” 3 days ago Up 3 days 0.0.0.0:80->80\/tcp prod-wordpress<\/em><\/p><\/blockquote>\n

At first glance, everything looked healthy the container was up there were no restarts Ports were mapped correctly.<\/p>\n

I checked and relaoded the webiste again just to find out the website was still down.<\/p>\n

Next Move –<\/h1>\n

Next I checked the POD utilisation<\/p>\n

$ docker stats prod-wordpress
\nCONTAINER ID \u00a0 \u00a0 \u00a0 \u00a0 \u00a0NAME \u00a0 \u00a0 \u00a0 \u00a0 \u00a0CPU % \u00a0 \u00a0 \u00a0 \u00a0 \u00a0MEM USAGE \/ LIMIT \u00a0 \u00a0 \u00a0 \u00a0 \u00a0MEM % \u00a0 \u00a0 \u00a0 \u00a0 \u00a0NET I\/O\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 BLOCK I\/O \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0PIDS
\n7f3a9c1b2d91\u00a0 \u00a0 prod-wordpress \u00a0 \u00a0 0.00%<\/strong> \u00a0 \u00a0 \u00a0 \u00a0 \u00a0110MiB \/ 2GiB \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a05.37% \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a00B \/ 0B \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a00B \/ 0B<\/strong> \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a01<\/p><\/blockquote>\n

Red flags:<\/p>\n

    \n
  • 0% CPU<\/li>\n
  • 0 network traffic<\/li>\n
  • Only 1 PID<\/li>\n<\/ul>\n

    Website was down but container showed \u201cUp\u201d
    \nThis clearly meant -> The container was running, but not serving any traffic.<\/strong><\/p>\n

     <\/p>\n

    Quick Fix –<\/h1>\n

    My Attempt of Quick Fix (The first thought of every DevOps) –<\/p>\n

    I tried restarting the container.<\/p>\n

    $ docker restart prod-wordpress<\/p><\/blockquote>\n

    Guess what it was hung<\/p>\n

    Next tried to stop the container<\/p>\n

    $ docker stop prod-wordpress<\/p><\/blockquote>\n

    It also hung, now it was a panic moment for me.<\/p>\n

    Even a force kill didn\u2019t work:<\/p>\n

    $ docker kill prod-wordpress<\/p><\/blockquote>\n

    I thought of restarting the Docker daemon (systemctl restart docker) might have been the easiest fix. But this was production all conatiners would be impacted and there would be unplanned downtime.<\/p>\n

    At this point of time I was pretty clueless. Then I went to seek help with online.<\/p>\n

    Docker-Shim<\/p>\n

    While researching similar incidents, I came to know about docker shim which is present in old docker versions.<\/p>\n

    docker-shim<\/p><\/blockquote>\n

    Older Docker versions used a helper process called docker-shim, which acted as an intermediate between Docker and the container runtime.<\/p>\n

    Each container had its own docker-shim process.<\/p>\n

    So I checked:<\/p>\n

    $ ps aux | grep docker-shim | grep prod-wordpress
    \nroot \u00a0 \u00a0 \u00a0 \u00a0 \u00a024791 \u00a0 \u00a0 \u00a0 \u00a0 \u00a00.0 0.1 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0123456 \u00a0 \u00a0 \u00a0 \u00a0 \u00a03456 ? Sl\u00a0 \u00a0 Feb08 0:01 \u00a0docker-shim -namespace moby -id 7f3a9c1b2d91 -address \/run\/docker\/libcontainerd\/docker-containerd.sock<\/p><\/blockquote>\n

    Findings –<\/p>\n

    docker-shim PID: 24791
    \nIt was the parent process of the container\u2019s main PID (24873)<\/p>\n

    Since docker-shim is just a helper process, I decided to kill it directly.<\/p>\n

    $ kill -9 24791<\/p><\/blockquote>\n

    Immediately checked Docker again:<\/p>\n

    $ docker ps -a
    \nCONTAINER ID\u00a0 \u00a0 \u00a0IMAGE\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 STATUS\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0NAMES
    \n7f3a9c1b2d91\u00a0 \u00a0 \u00a0 \u00a0 wordpress:6.4-apache\u00a0 \u00a0 Exited (137) 2 seconds ago.<\/strong>\u00a0 \u00a0prod-wordpress<\/p><\/blockquote>\n

    Bingo! The Zombie process was killed and container was in Exited state now<\/p>\n

    Now I started the container and it worked instantly:<\/p>\n

    $ docker start prod-wordpress
    \nprod-wordpress<\/p><\/blockquote>\n

    Verified traffic:<\/p>\n

    $ docker stats prod-wordpress
    \nCONTAINERID \u00a0 \u00a0 \u00a0 \u00a0 \u00a0NAME \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0CPU % \u00a0 \u00a0 \u00a0 \u00a0 \u00a0MEM USAGE \/ LIMIT \u00a0 \u00a0 \u00a0 NET I\/O
    \n7f3a9c1b2d91\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 prod-wordpress \u00a0 \u00a0 \u00a0 \u00a0 \u00a01.23%<\/strong> \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 145MiB \/ 2GiB \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a08.4MB \/ 6.9MB<\/strong><\/p><\/blockquote>\n

    Website was back online.<\/p>\n

    \"website-up\"

    website-up<\/p><\/div>\n

     <\/p>\n

    Learned from This Incident<\/h1>\n

    Containers Can Lie
    \nA container showing \u201cUp\u201d doesn\u2019t mean it\u2019s healthy or serving traffic.
    \ndocker-shim Was a Critical Link
    \ndocker-shim acted as the parent process
    \nIf it hung, the container lifecycle was broken<\/p>\n

    So next time if you face similar issues dont forget to check the docker shim process.<\/h1>\n","protected":false},"excerpt":{"rendered":"

    In this BLOG I will share an incident that taught me how containers really work under the hood. Production Down – Once I received production website down alert for one of my customer. As I checked the website was giving 502 Initial Checks – I immediately logged in to the production host to investigate. The […]<\/p>\n","protected":false},"author":2171,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":26},"categories":[5877],"tags":[1892,1883,3979,1785],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/77595"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/2171"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=77595"}],"version-history":[{"count":8,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/77595\/revisions"}],"predecessor-version":[{"id":77940,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/77595\/revisions\/77940"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=77595"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=77595"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=77595"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}