Who knew that ignoring the status of your inodes could bring down your Linux server? I can’t say the possibility crossed my mind all that often. Until, that is, ignoring the status of my inodes actually brought down my server.
First though, just what is an inode? It’s an object used by Unix systems to identify the disk location and attributes of files within a filesystem. There will usually be exactly one inode for each file or directory (the exception being hard linked files that can share a single inode).
You can see the inode numbers for each of the objects within a directory by running ls -i
$ ls -i
59776347 AsciidocFX
55312398 Desktop
55312402 Documents
55312399 Downloads
55312390 examples.desktop
56628996 git
55312403 Music
55312404 Pictures
55312401 Public
55312405 Videos
56236523 VirtualBox VMs
$
Now here’s the tricky bit. Just like the available disk space determines the upper limit on the files you can create or download to a computer, the number of files you can have is similarly capped by the an inode limit. To see your limit, run df -i and check out the value for your root directory under the Inodes column
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 60547072 628017 59919055 2% /
$
In the case of my home workstation, I’ve got more than 60 million inodes in total, of which only 2% are being used. No trouble there.
But when I logged into my server to run a security update the other day, I was hit with this:
dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-100_3.13.0-100.147_all.deb (--unpack):
unable to create `/usr/src/linux-headers-3.13.0-100/drivers/staging/dgap/Kconfig.dpkg-new' (while processing `./usr/src/linux-headers-3.13.0-100/drivers/staging/dgap/Kconfig'): No space left on device
Note the “No space left on device” warning. No space? I ran df and saw that there were still plenty of free GBs on hand. Just to be sure, I tried to create a new file in my home directory (to rule out some kind of permissions problem). No luck there:
$ touch ~/newfile
touch: cannot touch ‘/home/ubuntu/newfile’: No space left on device
I’ll admit that I didn’t immediately think to look at the inodes – as I said above: who knew…? But a quick Google search revealed that I wasn’t the first guy to get hit with this. Suitably informed (and embarrassed) I remembered that df -i will display file system inode usage, and here’s what it looked like for me:
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 524288 524288 0 100% /
IFree = 0, IUsed% = 100%. Not good. But what could possibly be generating enough files to drain my inode supply? Tracking down the directories containing the most files (i.e., using up the most inodes) can be done using some variation of this find command run from your root directory:
$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
But there’s a catch. Find saves its raw data in a temporary file…but it can’t do that as long as you have no free inodes. So you’ll have to delete one or two unneeded files just to get yourself going.
At any rate, with that out of the way, find from my root directory returned this:
$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
5 root
48 tmp
127 sbin
128 bin
377 boot
989 etc
2888 home
6578 var
15285 lib
372893 usr
The problem obviously lay somewhere within the /usr directory. So I moved down to /usr and ran find once again:
$ cd usr
This time the guilty finger pointed straight to /usr/src:
$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
6 include
160 sbin
617 bin
7211 lib
16518 share
348381 src
So just what goes on in /usr/src? Well it turns out that that’s where the kernel headers are kept. All kernel headers, including those left over from previous kernel versions installed on your machine. If you drill down through the directory tree, you’ll see that there are, indeed, a great many files. Mountains of ’em, in fact.
To free up some space, you may need to manually remove some of the older directories. Then, assuming you’re using a Debian/Ubuntu distribution, let dpkg safely remove whatever else isn’t needed through –configure:
$ sudo dpkg --configure -a
Then, to safely remove all old headers, run autoremove:
$ sudo apt-get autoremove
And everything should be back to optimal working order. The moral of the story? Details matter.
