If you're using AWS EKS and for some reason you get:
no space left on device: unknown
Check if:
- you're using XFS
- your disk usage is < 85% (At 85%, k8s triggers GC)
- you're pulling large 2GB docker images OR docker images with node_modules with hundreds of thousands of files
Then consider inode contiguous fragmentation issue:
https://support.microfocus.com/kb/doc.php?id=7014318
Userland (hacky) Solution:
Tune this to garbage collect more a: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/
- imageGCHighThresholdPercent
- imageGCLowThresholdPercent
Kernel (proper) Solution:
- man mkfs.xfs
- inode-options
- sparse (enable? is this a performance hit?)
- inode-options
- ikeep vs nokeep: (Some other configuration about not unallocating inodes after file deletion.. some reusage thing? Maybe not..)
For diagnostics look at the before and after from a crictl rmi --prune
run..:
xfs_db -r -c 'freesp -s' ${mount_name}
crictl rmi --prune
xfs_db -r -c 'freesp -s' ${mount_name}
And see how the contiguous inode histogram changes