Who knew that ignoring the status of your inodes could bring down your Linux server? I can’t say the possibility crossed my mind all that often. Until, that is, ignoring the status of my inodes actually brought down my server.
First though, just what is an inode? It’s an object used by Unix systems to identify the disk location and attributes of files within a filesystem. There will usually be exactly one inode for each file or directory (the exception being hard linked files that can share a single inode).
You can see the inode numbers for each of the objects within a directory by running ls -i
$ ls -i 59776347 AsciidocFX 55312398 Desktop 55312402 Documents 55312399 Downloads 55312390 examples.desktop 56628996 git 55312403 Music 55312404 Pictures 55312401 Public 55312405 Videos 56236523 VirtualBox VMs $
Now here’s the tricky bit. Just like the available disk space determines the upper limit on the files you can create or download to a computer, the number of files you can have is similarly capped by the an inode limit. To see your limit, run df -i and check out the value for your root directory under the Inodes column
$ df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 60547072 628017 59919055 2% / $
In the case of my home workstation, I’ve got more than 60 million inodes in total, of which only 2% are being used. No trouble there.
But when I logged into my server to run a security update the other day, I was hit with this:
dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-100_3.13.0-100.147_all.deb (--unpack): unable to create `/usr/src/linux-headers-3.13.0-100/drivers/staging/dgap/Kconfig.dpkg-new' (while processing `./usr/src/linux-headers-3.13.0-100/drivers/staging/dgap/Kconfig'): No space left on device
Note the “No space left on device” warning. No space? I ran df and saw that there were still plenty of free GBs on hand. Just to be sure, I tried to create a new file in my home directory (to rule out some kind of permissions problem). No luck there:
$ touch ~/newfile touch: cannot touch ‘/home/ubuntu/newfile’: No space left on device
I’ll admit that I didn’t immediately think to look at the inodes – as I said above: who knew…? But a quick Google search revealed that I wasn’t the first guy to get hit with this. Suitably informed (and embarrassed) I remembered that df -i will display file system inode usage, and here’s what it looked like for me:
$ df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda1 524288 524288 0 100% /
IFree = 0, IUsed% = 100%. Not good. But what could possibly be generating enough files to drain my inode supply? Tracking down the directories containing the most files (i.e., using up the most inodes) can be done using some variation of this find command run from your root directory:
$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
But there’s a catch. Find saves its raw data in a temporary file…but it can’t do that as long as you have no free inodes. So you’ll have to delete one or two unneeded files just to get yourself going.
At any rate, with that out of the way, find from my root directory returned this:
$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n 5 root 48 tmp 127 sbin 128 bin 377 boot 989 etc 2888 home 6578 var 15285 lib 372893 usr
The problem obviously lay somewhere within the /usr directory. So I moved down to /usr and ran find once again:
$ cd usr
This time the guilty finger pointed straight to /usr/src:
$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n 6 include 160 sbin 617 bin 7211 lib 16518 share 348381 src
So just what goes on in /usr/src? Well it turns out that that’s where the kernel headers are kept. All kernel headers, including those left over from previous kernel versions installed on your machine. If you drill down through the directory tree, you’ll see that there are, indeed, a great many files. Mountains of ’em, in fact.
To free up some space, you may need to manually remove some of the older directories. Then, assuming you’re using a Debian/Ubuntu distribution, let dpkg safely remove whatever else isn’t needed through –configure:
$ sudo dpkg --configure -a
Then, to safely remove all old headers, run autoremove:
$ sudo apt-get autoremove
And everything should be back to optimal working order. The moral of the story? Details matter.