TLUG Mailing List

On Thu, Jun 7, 2012 at 4:01 AM, Raymond Wan <rwan.kyoto@example.com> wrote:

> If fact, I dont have log files, but emails
The difference between e-mails and log files is just the e-mail header. :-)

Yes, plus some details, that I think you include in "headers" (footer, and difference between the header recipient and email header),
but easy to do, anyway :-)

If diskspace isn't an issue, keep the original, unfiltered log somewhere.
You may need it again some day.

Yes, you are right, and nobody (I suppose) will say the contrary!

As said before, I think, I can rerun the script without issue. This is a personal backup.
And it takes 10 minutes to run it again (to be frank, maybe 12-24 hours if I remove an
option -> if I want a full backup from scratch, ignoring previous ones).
Yes, I could keep the last log somewhere (or surely a few of them). This was my first used solution in
fact (not using cron's email facility, but creating first a log file, then sending it).
It is surely the best solution in a company... And I would *never, *never** do what I want to do for my
home backup in any serious environment (and never did in my life, I was the first to shout for "logs, more logs,
why we don't know what happened in the sell XX at 1.01 between 11:11:534 and 11:11:545".)

Different users, different needs :-) Some need microsecond (I did), some seconds (I did), some minutes,
etc... Even ten years logs to produce...

Could I say:
Could a same TLUG user have different different situations (times), different needs, then different logs needs,
without flames ? I know your *own* backup *is* not at same level as your company's.

> After all this is just my home dir backup. Nothing really vital. And I believe incremental backups would better give information on what is really changed (instead of everything, including what is not changed, which is the rsync works with my options). This is what my grep is supposed to do (if I did not make a mistake :-)

For your incremental backups, you're creating only one backup or
you're keeping several? Sometimes, you don't know something went
wrong or a file is missing until after a few backups have taken place.
So, it is good to have the old ones around, too.

In fact, as I use --link-dest option of rsync, so we cannot call it a real incremental
backup (I specify a directory in server - the source - and files unchanged are hard-linked
to this dir. New ones are created in new dir, old one disappear in new dir).
This allows to have a *full* fackup in each directory (daily-1, 2, 3, etc..., up to yearly-1, 2, etc...).
A file will really disappear on the server when the last hard link to a file disappear.
I have some files on my server with 23 hard links (and maybe more, I did not check deeply). It means
they are *really* physically in 23 backup dirs in the server. If I remove any of them, 22 links will remain (* below).

So, again, this is nor a real full backup (as there are no copies of files), nor an incremental one (as every
backup gets a *full* copy of what is available at this moment on source disk).
Is there a name for that kind of backup?

And I think human error is more common than hardware failure (well,
depends on how sleepy you typically are and where you get your
hardware from :-) ). I often have to recover a file I deleted by
accident... :-D

We totally agree! If anybody this list never did remove any file by mistake, I would be curious about it.
Just to make everybody humble ;-)
You should start a thread about it, seriously... Mine arrived in the first day I was "alone" to manage
a trading room servers, a little than twenty years ago...

To come back to our subject: to remove 23 versions of the file on the backup server could either
happen by a voluntary choice (I really want to have a file disappear from archives, and this is fine for me, as it is personal),
or by a strong mistaken rm on the server itself, where all my directories would disappear over there.
I could add fire, etc... Shouganai in there (**).

Sorry, there are 2 longs (*) and (**) below. You probably do not need to read them.

Thanks,

Bruno.

(*) Example: I have files with same name with different of links. For instance a file being available since
a few days... I made a (real) example:
I did "ls -il */UNIQUE.out" on server side. A file I do not care. I created it on June 5.
First "ls", server side:
1660965934 -rw-rw-r--    3 br       hdusers   2196358 Jun 5 15:29 daily-01/UNIQUE.out
1660965934 -rw-rw-r--    3 br       hdusers   2196358 Jun 5 15:29 daily-02/UNIQUE.out
1660965934 -rw-rw-r--    3 br       hdusers   2196358 Jun 5 15:29 daily-03/UNIQUE.out
Second, after forcing a daily backup:
1660965934 -rw-rw-r--    4 br       hdusers   2196358 Jun 5 15:29 daily-01/UNIQUE.out
1660965934 -rw-rw-r--    4 br       hdusers   2196358 Jun 5 15:29 daily-02/UNIQUE.out
1660965934 -rw-rw-r--    4 br       hdusers   2196358 Jun 5 15:29 daily-03/UNIQUE.out
1660965934 -rw-rw-r--    4 br       hdusers   2196358 Jun 5 15:29 daily-04/UNIQUE.out
Third, after removing file on my home dir, and backup again:
1660965934 -rw-rw-r--    4 br       hdusers   2196358 Jun 5 15:29 daily-02/UNIQUE.out
1660965934 -rw-rw-r--    4 br       hdusers   2196358 Jun 5 15:29 daily-03/UNIQUE.out
1660965934 -rw-rw-r--    4 br       hdusers   2196358 Jun 5 15:29 daily-04/UNIQUE.out
1660965934 -rw-rw-r--    4 br       hdusers   2196358 Jun 5 15:29 daily-05/UNIQUE.out
This file will be moved everyday in later and later backups, still available... And no space used used since
version one... In fact I feel this is very "smart". There are surely some drawbacks, but I did not find any yet
(except of course manually change the file on the server itself)...

(**) In fact I planned a couple of years ago to have a kind of equivalent system (cross backup between
my own backup server and my father's one). My motivation was mostly about earthquakes (I was
in Japan at that time). He even bought the machine, also a raid-5. The idea was to send encrypted
versions of files abroad, with also all versions kept, we both had space for it.
For some familial reasons (that I won't explain here), I decided not to finish the implementation, it was
not important enough :-(

--

Re: [tlug] But too much logs kills the logs: How to Grok Logs