[tlug] Mysterious LVM issue

["Birkir A. Barkarson" <birkirb@example.com>]

If anyone has some experience with LVM I encountered some rather
mysterious and spooky things when using it.

In short I was copying about 216GB from one HD to another which had an
LVM VG with exactly 216GB free. The copy (cp -a) went well and except
for some bad data on the original nothing was reported. Once done though
the LVM drive reported 0 bytes free, 216GB used. Rather unexpected as I
would have thought it would either have a few bytes left or it would
have given me an error messages about the drive being full.

Then I tried to copy some more data onto the drive I finally got a drive
full message. And that's when things got interesting.  See the attached
 transcript for a log of what happened.

All of a sudden it was reporting a different size and generally doing
strange things until I had removed some data.  Thinking that surely some
data would have corrupted since the drive filled up during the original
copy I used 'find' to search for 0 byte files, ran a fsck on the drive,
did syncs to copies of the data I had, so far nothing seems to be broken.

Does anyone know if LVM is doing some voodoo behind the scenes that
might lead to this situation. I'm a little uneasy about the whole thing
so any wise words would be appreciated.

Cheers,
Birkir

birkirb@example.com df -h
Filesystem             Size   Used  Avail Use% Mounted on
/dev/hda2               11G   1.2G   8.9G  12% /
tmpfs                   66M      0    66M   0% /lib/init/rw
udev                    11M    66k    11M   1% /dev
tmpfs                   66M      0    66M   0% /dev/shm
/dev/hda1              104M    16M    83M  16% /boot
/dev/mapper/america_main-store
                       232G   230G      0 100% /srv
/dev/mapper/america_main-home
                       983M    25M   909M   3% /home
/dev/mapper/america_main-local
                       3.0G   473M   2.4G  17% /usr/local

<deleted about 8GB of stuff>

birkirb@example.com df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda2             9.9G  1.1G  8.3G  12% /
tmpfs                  62M     0   62M   0% /lib/init/rw
udev                   10M   64K   10M   1% /dev
tmpfs                  62M     0   62M   0% /dev/shm
/dev/hda1              99M   15M   79M  16% /boot
/dev/mapper/america_main-store
                      216G  211G     0 100% /srv
/dev/mapper/america_main-home
                      938M   24M  867M   3% /home
/dev/mapper/america_main-local
                      2.8G  451M  2.2G  17% /usr/local

<Deleted about 2GB of stuff>

birkirb@example.com df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda2             9.9G  1.1G  8.3G  12% /
tmpfs                  62M     0   62M   0% /lib/init/rw
udev                   10M   64K   10M   1% /dev
tmpfs                  62M     0   62M   0% /dev/shm
/dev/hda1              99M   15M   79M  16% /boot
/dev/mapper/america_main-store
                      216G  209G     0 100% /srv
/dev/mapper/america_main-home
                      938M   24M  867M   3% /home
/dev/mapper/america_main-local
                      2.8G  451M  2.2G  17% /usr/local

<Deleted about 7GB of stuff>

birkirb@example.com:/srv/store/store/video/tv/True Blood$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda2             9.9G  1.1G  8.3G  12% /
tmpfs                  62M     0   62M   0% /lib/init/rw
udev                   10M   64K   10M   1% /dev
tmpfs                  62M     0   62M   0% /dev/shm
/dev/hda1              99M   15M   79M  16% /boot
/dev/mapper/america_main-store
                      216G  202G  2.9G  99% /srv
/dev/mapper/america_main-home
                      938M   24M  867M   3% /home
/dev/mapper/america_main-local
                      2.8G  451M  2.2G  17% /usr/local