{"id":2292,"date":"2015-10-08T18:12:03","date_gmt":"2015-10-09T01:12:03","guid":{"rendered":"http:\/\/www.wellgolly.com\/?p=2292"},"modified":"2016-03-10T17:25:40","modified_gmt":"2016-03-11T01:25:40","slug":"disk-space","status":"publish","type":"post","link":"https:\/\/www.wellgolly.com\/?p=2292","title":{"rendered":"Disk space"},"content":{"rendered":"<p>I was asked to take a look at a server that was no longer responding. The first thing I did after going to the co-lo and booting the console in single user repair mode was look at disk usage.<br \/>\n<pre><code class=\"preserve-code-formatting\">\n$ df\nFilesystem&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1K-blocks&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Used Available Use% Mounted on\n\/dev\/sda1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 70557052&nbsp;&nbsp;70557052&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 100% \/\nnone&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;508792&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 224&nbsp;&nbsp;&nbsp;&nbsp;508568&nbsp;&nbsp; 1% \/dev\nnone&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;513040&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;513040&nbsp;&nbsp; 0% \/dev\/shm\nnone&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;513040&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1076&nbsp;&nbsp;&nbsp;&nbsp;511964&nbsp;&nbsp; 1% \/var\/run\nnone&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;513040&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;513040&nbsp;&nbsp; 0% \/var\/lock\nnone&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;513040&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;513040&nbsp;&nbsp; 0% \/lib\/init\/rw\n\/dev\/sdb1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 70557052&nbsp;&nbsp;46954944&nbsp;&nbsp;20018012&nbsp;&nbsp;71% \/media\/Backup\n<\/code><\/pre><\/p>\n<p>As you can see the server usage went to 100%. Normally it is near the size of the hot backup drive. So some file suddenly got large. When this happens, none of the services will respond. So you can\u2019t ssh in and the websites stop responding. <\/p>\n<p>Then I started looking for files that could be causing the problem. I was told that the last time this happened the log files weren\u2019t being erased. I check on this server from time to time and this week it was at 82%, so that didn\u2019t seem likely but I ran this command anyway.<br \/>\n<pre><code class=\"preserve-code-formatting\">\nsudo find \/var\/log\/ -size +20M | xargs du | sort -n -r | less\n<\/code><\/pre><br \/>\nNothing really jumped out at me but there were lots of old logs that had been zipped, and some that were labelled &#8216;old&#8217;, so I ran this to get rid of them<br \/>\n<pre><code class=\"preserve-code-formatting\">\nrm *.gz\nrm *.old\n<\/code><\/pre><\/p>\n<p>It made a little improvement, but not enough to get the df command to go below 100%. It did go to 68852672 which was enough to get ssh into the server after a reboot. I kept looking for the source of the large files but had no luck. We have a call in to the guy who fixed it last time hoping he remembers what he did.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I was asked to take a look at a server that was no longer responding. The first thing I did after going to the co-lo and booting the console in single user repair mode was look at disk usage. $ df Filesystem&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1K-blocks&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Used Available Use% Mounted on \/dev\/sda1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 70557052&nbsp;&nbsp;70557052&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 100% \/ none&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;508792&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 224&nbsp;&nbsp;&nbsp;&nbsp;508568&nbsp;&nbsp; 1% &hellip; <a href=\"https:\/\/www.wellgolly.com\/?p=2292\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Disk space<\/span><\/a><\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[],"class_list":["post-2292","post","type-post","status-publish","format-standard","hentry","category-computers"],"_links":{"self":[{"href":"https:\/\/www.wellgolly.com\/index.php?rest_route=\/wp\/v2\/posts\/2292","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wellgolly.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wellgolly.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wellgolly.com\/index.php?rest_route=\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wellgolly.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2292"}],"version-history":[{"count":0,"href":"https:\/\/www.wellgolly.com\/index.php?rest_route=\/wp\/v2\/posts\/2292\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wellgolly.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2292"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wellgolly.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2292"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wellgolly.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2292"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}