Re: [tlug] Overloaded mail spool causing memory instability?

["Stephen J. Turnbull" <stephen@example.com>]

Dave M G writes:

 > The question was if a large amount of email stored on the system can 
 > create instability.

The question I answered was the one you asked explicitly: "does having
a large mailbox use up memory?"  The answer is no, for the reasons I
gave.  [Curt points out that the append may be more complicated, but
AFAICS it still takes extra memory in proportion to the size of the
message, not in proportion to the size of the mbox on disk.]

Obviously it uses up space on the hard drive, but except in *really*
demanding applications (the kind of application where you care about a
handful of nanoseconds from matching disk requests to rotation speed),
that doesn't matter either, until you actually run out of allocatable
space.

 > If I leave a domain open to receive un-routed email, is it possible for 
 > some maximum amount of incoming email to cause instability? Like if I 
 > get few dozen, hundred, or thousand all at once?

The following resources allocated on demand are involved in receiving
mail:

1.  Network bandwidth
2.  Memory for the TCP/IP connection structures and transfer buffers
3.  TCP ports
4.  Interprocess communication sockets (I don't know if these are
    separate from 2 in Linux, but in theory they could be)
5.  CPU in interrupts kernel time slices for processing TCP
6.  Processes or threads for the MTA, milters, delivery agents (like
    procmail) and other stuff (spam & virus filters)
7.  "User-space" memory and CPU for the MTA and MDA-related processes
8.  Storage for the spooled mail

Maybe there are others, or people would like divide things up
differently, but that gives you an idea of the variety.

Shortage of any of them could result in delays or instability.

There are about 50,000 TCP ports available; you shouldn't run out of
those before something else brings the system to its knees.

Memory is almost always in short supply; if the rate of connections
causes you to start swapping virtual memory, you will experience
slowdowns and if it persists for more than a few seconds, instability.

I'm not sure about CPU, but it seems to me that performance generally
degrades gracefully if it's only CPU that's in contention (not RAM or
net bandwidth).

Storage for spooled mail is generally not a problem, but if you have
/var/mail on the same partition as say /tmp or /var/log, you can run
out of space, and serious trouble can break loose as your applications
start to fail to be able to log informational message or even errors,
etc.

 > I'm trying to find out what would be the root cause of the email 
 > instability I've seen recently. It seems potentially tied to the issue 
 > of un-routed email, but I can't quite grasp how.

The problem that I've seen is that you have a spammer hammering on
your server.  If you accept the message, you may need to wake up or
(much worse) start a dozen processes besides the MTA itself (virus
filters, spam checkers, the final delivery agent).  This uses up lots
of CPU, some memory, etc.  Worse, if you have a spam filter that
checks the source, you may need to make network connections, at least
to the DNS, perhaps several times.  I've seen greylisted spammers come
back once a second until you *do* accept the mail.  (I don't know
whether this is bad implementation, they just don't care, or a
deliberate retaliation for greylisting.)  That's a lot of load for
something you're going to unable to do anything useful with.

But if you refuse mail you can't route, you can save all those context
switches and other resources used by actually reading and processing
the mail text.

I don't know if this would lead to instability that a human could
percieve without log analysis, but why accept it at all?