ImageGCFailed

Errors

Source Log

Events:
  Type     Reason         Age                       From                      Message
  ----     ------         ----                      ----                      -------
  Warning  ImageGCFailed  4m14s (x15704 over 133d)  kubelet, banana-worker13  (combined from similar events): wanted to free 11841176371 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "cptactionhank/atlassian-jira-software:7.12.0" (must force) - container da502208489f is using its referenced image 1b29859343c2, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "mysql:5.7" (must force) - container 4d28acfd331e is using its referenced image 3147495b3a5c, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "docker-hub.foobar.work/foobar-dev/redis:amd64-7.4.0" (must force) - container 4bbe579d02fd is using its referenced image dae83f665c92]

Pretty Log

Events:
  Type     : Warning  
  Reason   : ImageGCFailed  
  Age      : 4m14s (x15704 over 133d)  
  From     : kubelet, banana-worker13  
  Message  : (combined from similar events): wanted to free 11841176371 bytes, but freed 0 bytes 
             space with errors in image deletion:
[
  rpc error: code = Unknown desc = Error response from daemon: 
      conflict: unable to remove repository reference 
      "cptactionhank/atlassian-jira-software:7.12.0" (must force) - 
      container da502208489f is using its referenced image 1b29859343c2,

  rpc error: code = Unknown desc = Error response from daemon: 
      conflict: unable to remove repository reference 
      "mysql:5.7" (must force) - 
      container 4d28acfd331e is using its referenced image 3147495b3a5c,

  rpc error: code = Unknown desc = Error response from daemon: 
      conflict: unable to remove repository reference 
      "docker-hub.foobar.work/foobar-dev/redis:amd64-7.4.0" (must force) - 
      container 4bbe579d02fd is using its referenced image dae83f665c92
]

• kubelet 尝试释放约 11GB 的磁盘空间；
• 失败原因是一些镜像正在被容器使用；
• 镜像必须 强制删除 才能清理，但容器仍在运行中或未正确终止。

Solution

Stop Containers

docker stop da502208489f 4d28acfd331e 4bbe579d02fd

Delete Containers

docker rm -f da502208489f 4d28acfd331e 4bbe579d02fd

Delete Images

docker rmi 1b29859343c2 3147495b3a5c dae83f665c92

Delete Images

docker rmi cptactionhank/atlassian-jira-software:7.12.0
docker rmi mysql:5.7
docker rmi docker-hub.foobar.work/foobar-dev/redis:amd64-7.4.0

清理未使用的镜像

如果不确定具体镜像，可以执行：

docker system prune -a

这个命令会删除所有未使用的镜像、容器和缓存文件，请谨慎使用。