Re: [tlug] formatting HDD leaves 11.7 GB untouched

[Curt Sampson <cjs@example.com>]

On 2019-12-02 10:48 +0900 (Mon), Thomas Blasejewicz wrote:

> I have a 160 GB HDD that was used in an old notebook running Mint 19.2,
> using only one partition.
> Since I changed the HDD, I would like to use this one for external storage.
> So .. I formatted the disk; the usual way; using ext4

What exactly is this "usual way"? If you're using a command-line
program, give us the exact command you used (including all arguments)
and its output.

Some background you may need:

The ext4 filesystem is generally in a partition, which will be a subset of
all the blocks on the disk. The `df` command is the correct tool to see
how much space is used and free in a filesystem:

    $ df /dev/mmcblk0p2
    Filesystem     1K-blocks  Used Available Use% Mounted on
    /dev/mmcblk0p2    241965 67571    161902  30% /boot
    $ echo $((241965-67571))
    174394

Note that the available amount of space is less than the total blocks
minus the used blocks. (In this case it's 12492 blocks less.) This is
because some of the blocks on the filesystem are used for metadata
storage such as the i-nodes that contain permission bits, timestamps,
and the block numbers in which the file data is stored.

Unless you're being very unsual (e.g., doing `mke2fs /dev/sda` instead
of `mke2fs /dev/sda0`, which is not recommended!) there will be
additional space on the whole disk that is not part of that partition
that contains the filesystem. This will be at least the boot block
area and partition table, perhaps some unused space, and perhaps even
other partitions. You can see this with `lsblk`:

    $ lsblk /dev/mmcblk0
    NAME                    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
    mmcblk0                 179:0    0 29.1G  0 disk
    ├─mmcblk0p1             179:1    0  512M  0 part  /boot/efi
    ├─mmcblk0p2             179:2    0  244M  0 part  /boot
    └─mmcblk0p3             179:3    0 28.4G  0 part
      └─mmcblk0p3_crypt     254:0    0 28.4G  0 crypt
          ├─iambic--vg-swap_1 254:1  0  2.8G  0 lvm   [SWAP]
          └─iambic--vg-root   254:2  0 25.6G  0 lvm   /

Note that this doesn't know anything about how much space is free in
any partition; that depends on the particular filesystem in that
partition. Some partitions don't even have a concept of "free space":
mmcblk0p3 is an encrypted partition which always "uses" the entire
partition, every encrypted block in it maps directly to an unencrypted
block in the "virtual" mmcblk0p3_crypt partition, regardless of what
any filesystems nested inside that consider any of those blocks used
or free.

> Afterwards, the properties showed: 8.1 GB used

Properties in what program? And was it showing information about the
filesystem in a partition, or partitions on a disk, or what?

> Looking at it with "disks" I get: 160 GB -> 156 GB free; 2.3% used.

I can't find any program named "disks" in any Debian packages. Is this
perhaps a tab or something in a GUI program? Similarly, is it display
information about a filesystem in a partition, partitions on the disk,
or what?

> I formatted again, this time enabling "erase". That took close to 2 hours.
> Same result.

Right. The "erase" changed nothing except that without that mke2fs
just wrote its metadata; any blocks that it allocated as "free to
store data" would contain whatever old data happened to be there
before. (Those old data are overwritten as soon as that free block is
allocated to a new or expanding file.)

When you used the "erase" option it actually went around to every
block it initially allocated as "free" and wrote zeros to it, which is
why that took so long. But that makes no difference to how much free
space you have in the partition; it always starts entirely "empty,"
i.e. with the maximum amount of free space for data storage after the
metadata has been allocated.

> After googling I found the "d command" that is supposed to remove the MBR

In what program is this "d command"? At any rate, you don't want to
remove the MBR ("master boot record," which also includes the
partition table) unless you're going to put another one in there,
because this has bad effects that I won't get into here. (Catch me in
person and feed me a beer and I'll go into it in detail.) Always make
sure your disks have an MBR and put filesystems into appropriately
marked partitions.

> That means, if I understand that correctly, the disk still holds 11.7 GB of
> something that I do not need.
> Naturally, those 11 GB are not displayed as any sort of file of folder.

I'm not sure exactly what you're looking at, but likely that's 11.7 GB
of metadata. (11.7÷160 is about 7%, which is typical of filesystem
metadata overhead.) You need that metadata! Having the file data on
the disk somewhere is no good if the filesystem doesn't know where
it's stored on the disk and thus can't retrieve it.

cjs
-- 
Curt J. Sampson      <cjs@example.com>      +81 90 7737 2974

To iterate is human, to recurse divine.
    - L Peter Deutsch