[Tfug] disk usage mystery..

[Ian McEwen]

On Mon, Mar 05, 2012 at 08:36:57PM -0700, John Gruenenfelder wrote:
> On Mon, Mar 5, 2012 at 20:07, JD Rogers <rogersjd at gmail.com> wrote:
> > Hi All,
> >
> > I've got a weird disk usage situation. I have about 1GB free on my
> > laptop at boot according to "df -h". After running for a day or so, I
> > get a friendly ubuntu warning that I have only 20MB free and "df -h"
> > reports 100% used (I have ext4 tuned to keep only 2GB for root, but
> > user space is filled). The tricky bit is that I can't see where this
> > disk space is going.
> >
> > Since I noticed a reboot will reset back to about 1GB free, I
> > suspected something in /var like logs or maybe tmp or cache files in
> > the user home dir. However, "du -h --max-depth=1" in any of these
> > suspect places turned up no more than a few MG difference from at boot
> > time. In fact, I even did "du -h --max-depth=1 > duatboot" and waited
> > a until my disk was full, did it again and diffed the outputs. To my
> > surprised, the entire disk is essentially the same. So where is the
> > disk usage going!?
> >
> > Now granted, 1GB free is not ideal these days, but I go through a lot
> > of data and only move it off to my workstation when I need to free up
> > space or move to another project. Running around 1GB free has worked
> > fine for the past year or so, and this behavior has only showed up in
> > the past few weeks.
> >
> > Anyone have any insights or suggestions?
> >
> > ubuntu 11.10 updated
> > ext4, encrypted home with ecryptfs
> > swap is on a separate partition
>
> JD,
>
> This is just a shot in the dark, but perhaps you have some broken
> process (either a bug or something more malicious) that is filling up
> the space with temp files.  It is not uncommon for a process to create
> a random temp file, open it, then unlink it.  At this point, the
> process can still use the file for storage but nobody else should be
> able to open the file because it will not be listed on the file
> system.  As soon as the process closes the file, the final link is
> released and the data is finally deleted.
>
> This would explain why you can't find the files and why it gets better
> after a reboot.  However, I do not know how du treats these sorts of
> files when printing directory space usage.  If this is indeed the
> case, I'm not sure how easy it will be to track down the culprit.
> Normally, temp files, and especially those created in this manner,
> hold very little data.
>

Presumably 'lsof | grep "(deleted)"' when disk usage is high will show
such files (and, conveniently, the processes with the open links). I had
a similar disk-usage problem situation recently (fixed by changing a
script's configurable tmpdir), but didn't make the connection.

On my system I get a number of false positives here that actually exist
(e.g. /usr/bin/xinit, which clearly still exists on my system!).
However, it's pretty easy to check if you can see a given file.

I know from experience that df will see this disk usage; du may not,
since it's file-oriented rather than partition-oriented. You might also
check if du and df differ substantially in their numbers; that might
suggest John is on the right track.

--
Ian McEwen <ianmcorvidae at ianmcorvidae.net> <ihm08 at hampshire.edu>
A262 D5C4 40CB 0E1C 5F24 C3A1 ABED 1ABD 7131 A76F
http://ianmcorvidae.net/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://tfug.org/pipermail/tfug_tfug.org/attachments/20120305/9c6fa9eb/attachment.sig>