[tlug] Re: rsync efficiency (was: The Mother of All (bash) Commands)

["Jim Breen" <jimbreen@example.com>]

Attila Kinali <attila@example.com> pounded the table and roared:
>  Curt Sampson <cjs@example.com> wrote:
>  > On 2008-03-18 15:55 +0100 (Tue), Attila Kinali wrote:
>  >
>  > > Yes, rsync is horribly inefficient in nearly all ways.
>  > > ...(ie, absolutely CPU limited).
>  >
>  > It's actually extremely efficient for the situation it was designed for,
>  > which is a large amount of very similar data on two machines and a slow
>  > network pipe.
>
>  Unless this data is contained in a lot of small files, in which
>  case rsync's very stupid behaviour of traversing the file/directory
>  tree more than once becomes a very expensive operation.

A bit like using Excel to perform Fourier transforms.

rsync was designed to be efficient in synchronizing large files with
sparse changes. Using it on masses of small files is simply using the
wrong tool. You may as well ftp a tarball.

>  I'm backuping my mails which are in an maildir to another machine.
>  Each backup takes about an hour, in which rsync does nothing
>  but stat files on both sites. In the end, there is about 5min
>  of transfering the new data.. that's all.

The good old mbox format was great for rsyncing.

>  Of course, my example is a bit on the extreme side, but rsync
>  claims to be fast and efficient, which it definitly is not.

I use rsync for updating the WWWJDIC mirrors. Each mirror updates about
100Mb of dictionary files each day, of which only a few kb have changed.
The file updates usually only take a few seconds. I use make to create
a manifest of the files which have changed. The mirror retrieves the
manifest then rsyncs the changed files. All very fast and efficient.

Cheers

Jim
-- 
Jim Breen
Honorary Senior Research Fellow
Clayton School of Information Technology,
Monash University, VIC 3800, Australia
http://www.csse.monash.edu.au/~jwb/