[Tfug] disk usage mystery..

[JD Rogers]

Narrowed down the problem.. lightdm or at least something related to X.

If I switch to a virt term and "stop lightdm" the disk is still full,
but ps shows that lightdm doesn't fully exit because of pulseaudio. So
I "killall pulseaudio" and bam! I have 1GB free. No reboot needed and
its not a system/kernel bug. Still no sign of what file is actually
taking up 1GB of space, even using lsof. FWIW, killing pulseaudio
without shutting down x and stopping lightdm has no affect.

On a whim, I tried using gdm instead of lightdm and got all excited
because it appeared to work fine for a day, but I see my free disk
space is plummeting at a rate of about 3MB every minute as I type
this.

JDR

On Thu, Mar 8, 2012 at 8:34 AM, JD Rogers <rogersjd at gmail.com> wrote:
> On Tue, Mar 6, 2012 at 8:32 AM, JD Rogers <rogersjd at gmail.com> wrote:
>>> If you have the time, I would suggest rebooting (or using telinit to
>>> switch) into a single user run level.  Then you should just have the
>>> kernel, init, and your shell running.  If it is *still* leaking, then
>>> it seems that perhaps you have a kernel bug?
>>
>> Not a bad idea.. I may try that when I can.. probably wednesday.
>
> Haven't had a chance to wait a day in single user mode yet, but I did
> try "telinit 1" with interesting results. The disk was full but after
> entering runlevel 1, the disk had 900MB free.
>
> The disk was freed up even though switching to singleuser mode did not
> unmount the ecryptfs homedir.  I think the errors I was seeing from
> ecryptfs are unrelated to the disk usage issue. The fact that if I
> don't start firefox, those errors never arise points to that, but for
> a conclusive test I'll have to try letting the disk fill up while not
> ever starting FF.
>
> In any case, there certainly seems to be a process somewhere that is
> tying up diskspace, but the fact that it does not show up using lsof
> or du is very perplexing. I guess the next step will be waiting for a
> full disk and then killing daemons and processes recklessly until I
> see a change. Then maybe I'll try the no FF thing or maybe waiting in
> singleuser mode to see if the disk fills.
>
> Still a very strange issue.
>
>> [snip]
>>> Here's another shot in the dark: if it *is* in fact a kernel file
>>> system bug, then perhaps it is related the the ext4 journal?  I should
>>> think it has a relatively small maximum size, but maybe it is growing
>>> out of control and this data is released by an unmount?  Seems like
>>> you would encounter other effects from this, though.
>>>
>>> Another unlikely but possible culprit could be ecryptfs.  You didn't
>>> say if /home (or wherever the encrypted data is stored) is on its own
>>> partition.  If it is sharing space with root, maybe it's using up the
>>> extra space somehow?
>>
>> I'm leaning towards ecryptfs. Sheepishly, I must admit I didn't notice
>> this earlier.. I am apparently also getting bazillions of these in my
>> logs:
>> [ 8923.440844] Valid eCryptfs headers not found in file header region
>> or xattr region
>> [ 8923.440849] Either the lower file is not in a valid eCryptfs
>> format, or the key could not be retrieved. Plaintext passthrough mode
>> is not enabled; returning -EIO
>>
>> I found a few bug reports that mostly seem to be duplicates of
>> https://bugs.launchpad.net/ecryptfs/+bug/509180
>> but my symptoms are not really the same, and this might be totally
>> unrelated, because these errors stop when I close firefox. Yet the
>> disk usage does not resolve itself.
>>
>> It appears I have some more digging to do. At least I feel like I have
>> some things to try. Thanks for the suggestions!