{"id":179,"date":"2016-10-26T13:00:28","date_gmt":"2016-10-26T13:00:28","guid":{"rendered":"http:\/\/bootstrap-it.com\/blog\/?p=179"},"modified":"2017-01-09T15:24:48","modified_gmt":"2017-01-09T15:24:48","slug":"linux-inodes","status":"publish","type":"post","link":"https:\/\/bootstrap-it.com\/blog\/?p=179","title":{"rendered":"Ignore Your Inodes at Your Own Peril!"},"content":{"rendered":"
<\/a><\/a><\/a><\/a><\/a><\/div>

\"linux-inode\"Who knew that ignoring the status of your inodes could bring down your Linux server? I can’t say the possibility crossed my mind all that often. Until, that is, ignoring the status of my inodes actually brought down my<\/strong> server<\/a>.<\/p>\n

First though, just what is an inode? It’s an object used by Unix systems to identify the disk location and attributes of files within a filesystem. There will usually be exactly one inode for each file or directory (the exception being hard linked files that can share a single inode).<\/p>\n

You can see the inode numbers for each of the objects within a directory by running ls -i<\/p>\n

 <\/p>\n

\r\n$ ls -i\r\n59776347 AsciidocFX\r\n55312398 Desktop\r\n55312402 Documents\r\n55312399 Downloads\r\n55312390 examples.desktop\r\n56628996 git\r\n55312403 Music\r\n55312404 Pictures\r\n55312401 Public\r\n55312405 Videos\r\n56236523 VirtualBox VMs\r\n$\r\n<\/code><\/pre>\n
Now here’s the tricky bit. Just like the available disk space determines\u00a0the upper limit on the files you can create or download to a computer, the number<\/em> of files you can have is similarly capped by the an inode limit. To see your limit, run df -i and check out the value for your root directory under the Inodes column<\/p>\n
\r\n$ df -i\r\nFilesystem Inodes IUsed IFree IUse% Mounted on\r\n\/dev\/sda2 60547072 628017 59919055 2% \/\r\n$\r\n<\/code><\/pre>\n
In the case of my home workstation, I’ve got more than 60 million inodes in total, of which only 2% are being used. No trouble there.<\/p>\n
But when I logged into my server to run a security update the other day, I was hit with this:<\/p>\n
\r\ndpkg: error processing archive \/var\/cache\/apt\/archives\/linux-headers-3.13.0-100_3.13.0-100.147_all.deb (--unpack):\r\nunable to create `\/usr\/src\/linux-headers-3.13.0-100\/drivers\/staging\/dgap\/Kconfig.dpkg-new' (while processing `.\/usr\/src\/linux-headers-3.13.0-100\/drivers\/staging\/dgap\/Kconfig'): No space left on device\r\n<\/code><\/pre>\n
Note the “No space left on device” warning. No space? I ran df and saw that there were still plenty of free GBs on hand. Just to be sure, I tried to create a new file in my home directory (to rule out some kind of permissions problem). No luck there:<\/p>\n
\r\n$ touch ~\/newfile\r\ntouch: cannot touch \u2018\/home\/ubuntu\/newfile\u2019: No space left on device\r\n<\/code><\/pre>\n
I’ll admit that I didn’t immediately think to look at the inodes – as I said above: who knew…? But a quick Google search revealed that I wasn’t the first guy to get hit with this. Suitably informed (and embarrassed) I remembered that df -i will display file system inode usage, and here’s what it looked like for me:<\/p>\n
\r\n$ df -i\r\nFilesystem Inodes IUsed IFree IUse% Mounted on\r\n\/dev\/xvda1 524288 524288 0 100% \/\r\n<\/code><\/pre>\n
IFree = 0, IUsed% = 100%. Not good. But what could possibly be generating enough files to drain my inode supply? Tracking down the directories containing the most files (i.e., using up the most inodes) can be done using some variation of this find command run from your root directory:<\/p>\n
\r\n$ sudo find . -xdev -type f | cut -d \"\/\" -f 2 | sort | uniq -c | sort -n\r\n<\/code><\/pre>\n
But there’s a catch. Find<\/strong> saves its raw data in a temporary file…but it can’t do that as long as you have no free inodes. So you’ll have to delete one or two unneeded files just to get yourself going.<\/p>\n
At any rate, with that out of the way, find<\/strong> from my root directory returned this:<\/p>\n
\r\n$ sudo find . -xdev -type f | cut -d \"\/\" -f 2 | sort | uniq -c | sort -n\r\n5 root\r\n48 tmp\r\n127 sbin\r\n128 bin\r\n377 boot\r\n989 etc\r\n2888 home\r\n6578 var\r\n15285 lib\r\n372893 usr\r\n<\/code><\/pre>\n
The problem obviously lay somewhere within the \/usr directory. So I moved down to \/usr and ran find<\/strong> once again:<\/p>\n
\r\n$ cd usr\r\n<\/code><\/pre>\n
This time the guilty finger pointed straight to \/usr\/src:<\/p>\n
\r\n$ sudo find . -xdev -type f | cut -d \"\/\" -f 2 | sort | uniq -c | sort -n\r\n6 include\r\n160 sbin\r\n617 bin\r\n7211 lib\r\n16518 share\r\n348381 src\r\n<\/code><\/pre>\n
So just what goes on in \/usr\/src? Well it turns out that that’s where the kernel headers are kept. All kernel headers, including those left over from previous kernel versions installed on your machine. If you drill down through the directory tree, you’ll see that there are, indeed, a great many files. Mountains of ’em, in fact.<\/p>\n
To free up some space, you may need to manually remove some of the older directories. Then, assuming you’re using a Debian\/Ubuntu distribution, let dpkg safely remove whatever else isn’t needed through –configure:<\/p>\n
\r\n$ sudo dpkg --configure -a\r\n<\/code><\/pre>\n
Then, to safely remove all old headers, run autoremove:<\/p>\n
\r\n$ sudo apt-get autoremove\r\n<\/code><\/pre>\n
And everything should be back to optimal working order. The moral of the story? Details matter.<\/p>\n","protected":false},"excerpt":{"rendered":"
Who knew that ignoring the status of your inodes could bring down your Linux server? I can’t say the possibility crossed my mind all that often. Until, that is, ignoring the status of my inodes actually brought down my server.… Continue Reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":190,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-179","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"yoast_head":"\nIgnore Your Inodes at Your Own Peril! - Bootstrap IT<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bootstrap-it.com\/blog\/?p=179\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Ignore Your Inodes at Your Own Peril! - Bootstrap IT\" \/>\n<meta property=\"og:description\" content=\"Who knew that ignoring the status of your inodes could bring down your Linux server? I can’t say the possibility crossed my mind all that often. Until, that is, ignoring the status of my inodes actually brought down my server.… Continue Reading →\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bootstrap-it.com\/blog\/?p=179\" \/>\n<meta property=\"og:site_name\" content=\"Bootstrap IT\" \/>\n<meta property=\"article:published_time\" content=\"2016-10-26T13:00:28+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-01-09T15:24:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/bootstrap-it.com\/blog\/wp-content\/uploads\/linux-inode.png\" \/>\n\t<meta property=\"og:image:width\" content=\"664\" \/>\n\t<meta property=\"og:image:height\" content=\"800\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"David Clinton\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@davidbclinton\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"David Clinton\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/?p=179\",\"url\":\"https:\/\/bootstrap-it.com\/blog\/?p=179\",\"name\":\"Ignore Your Inodes at Your Own Peril! - Bootstrap IT\",\"isPartOf\":{\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#website\"},\"datePublished\":\"2016-10-26T13:00:28+00:00\",\"dateModified\":\"2017-01-09T15:24:48+00:00\",\"author\":{\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/2db6d8bce89bf20e8f06440c428abe68\"},\"breadcrumb\":{\"@id\":\"https:\/\/bootstrap-it.com\/blog\/?p=179#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/bootstrap-it.com\/blog\/?p=179\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/?p=179#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/bootstrap-it.com\/blog\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Ignore Your Inodes at Your Own Peril!\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#website\",\"url\":\"https:\/\/bootstrap-it.com\/blog\/\",\"name\":\"Bootstrap IT\",\"description\":\"Learn technology using technology\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/bootstrap-it.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/2db6d8bce89bf20e8f06440c428abe68\",\"name\":\"David Clinton\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/fa132defc2e7804e08ce17c4644b6f88c498d73ed8d4c9c72039189f90df1af8?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/fa132defc2e7804e08ce17c4644b6f88c498d73ed8d4c9c72039189f90df1af8?s=96&d=mm&r=g\",\"caption\":\"David Clinton\"},\"sameAs\":[\"http:\/\/bootstrap-it.com\/\",\"dbclinton\",\"https:\/\/twitter.com\/davidbclinton\"],\"url\":\"https:\/\/bootstrap-it.com\/blog\/?author=2\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Ignore Your Inodes at Your Own Peril! - Bootstrap IT","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bootstrap-it.com\/blog\/?p=179","og_locale":"en_US","og_type":"article","og_title":"Ignore Your Inodes at Your Own Peril! - Bootstrap IT","og_description":"Who knew that ignoring the status of your inodes could bring down your Linux server? I can’t say the possibility crossed my mind all that often. Until, that is, ignoring the status of my inodes actually brought down my server.… Continue Reading →","og_url":"https:\/\/bootstrap-it.com\/blog\/?p=179","og_site_name":"Bootstrap IT","article_published_time":"2016-10-26T13:00:28+00:00","article_modified_time":"2017-01-09T15:24:48+00:00","og_image":[{"width":664,"height":800,"url":"https:\/\/bootstrap-it.com\/blog\/wp-content\/uploads\/linux-inode.png","type":"image\/png"}],"author":"David Clinton","twitter_card":"summary_large_image","twitter_creator":"@davidbclinton","twitter_misc":{"Written by":"David Clinton","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/bootstrap-it.com\/blog\/?p=179","url":"https:\/\/bootstrap-it.com\/blog\/?p=179","name":"Ignore Your Inodes at Your Own Peril! - Bootstrap IT","isPartOf":{"@id":"https:\/\/bootstrap-it.com\/blog\/#website"},"datePublished":"2016-10-26T13:00:28+00:00","dateModified":"2017-01-09T15:24:48+00:00","author":{"@id":"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/2db6d8bce89bf20e8f06440c428abe68"},"breadcrumb":{"@id":"https:\/\/bootstrap-it.com\/blog\/?p=179#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bootstrap-it.com\/blog\/?p=179"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/bootstrap-it.com\/blog\/?p=179#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/bootstrap-it.com\/blog"},{"@type":"ListItem","position":2,"name":"Ignore Your Inodes at Your Own Peril!"}]},{"@type":"WebSite","@id":"https:\/\/bootstrap-it.com\/blog\/#website","url":"https:\/\/bootstrap-it.com\/blog\/","name":"Bootstrap IT","description":"Learn technology using technology","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bootstrap-it.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/2db6d8bce89bf20e8f06440c428abe68","name":"David Clinton","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bootstrap-it.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/fa132defc2e7804e08ce17c4644b6f88c498d73ed8d4c9c72039189f90df1af8?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/fa132defc2e7804e08ce17c4644b6f88c498d73ed8d4c9c72039189f90df1af8?s=96&d=mm&r=g","caption":"David Clinton"},"sameAs":["http:\/\/bootstrap-it.com\/","dbclinton","https:\/\/twitter.com\/davidbclinton"],"url":"https:\/\/bootstrap-it.com\/blog\/?author=2"}]}},"_links":{"self":[{"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/179","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=179"}],"version-history":[{"count":18,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/179\/revisions"}],"predecessor-version":[{"id":227,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/179\/revisions\/227"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=\/wp\/v2\/media\/190"}],"wp:attachment":[{"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=179"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=179"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bootstrap-it.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}