Re: [tlug] DDB/CJKV-E Web Host under DDOS attack

[Jim Breen <jimbreen@example.com>]

On 14 March 2016 at 06:18, Curt Sampson <cjs@example.com> wrote:
>
> Well, I had a bunch of commments, but then I came across this:
>
> On 2016-03-10 10:24 +1100 (Thu), Jim Breen wrote:
>
> > A rush of requests has lead to a heap of processes (httpd, etc.) being
> > spawned, a sudden runout of RAM and swap, and the system eventually
> > thrashes itself to death.
>
> That's your number one issue right there. In general, if you *ever*
> allow requests to cause new processes to be spawned, you've set yourself
> up to be hurt a lot more by a DDOS than you need to be.

Or by over-enthusiatic/badly-constructed SEO crawlers, etc. etc. Of course
it' hard to stop requests leading to processes in a typical web system. Our
current belt-and-braces approach, which is limit httpd numbers (currently set
at 50 max), and for the maintenance system stop the scripts calling the
database if free swap is below a threshold, seems to be holding the line
well.

> From your other comments (e.g., about wwjdic being in C and not using
> an external DBMS) I can see that the system is rather less typical than
> I'd first assumed, so any other advice before I properly understand it
> (including my advice in messages before this one) is going to be kind of
> hit-or-miss.
>
> That said, I find it hard to think of a situation where, for a site like
> yours, if properly written, a DDOS could run you out of CPU or disk
> before bandwith on "regular" machines (by which I mean, your typical
> cheap i7 things without 10GigE interfaces).

Yes, very high loads tend to show up as bandwidth usage. Last year
our main cloud server showed a big jump in bandwidth use. The logs
showed a heap of accesses from a client called "TRAG", all hitting
the text-glossing function in wwwjdic. It turns out it's a system called
"Translation Aggregator" and it can be configured to call wwwjdic
automatically for every Japanese string - small wonder I was seeing
thousands of calls an hour from some IP addresses,

See: http://www.hongfire.com/forum/showthread.php/94395-Translation-Aggregator

Anyway I discussed it with the maintainer and he made some changes
which resulted in the load going down significantly. I still have
an alert on it, and I occasionally sin-bin IP addresses which
overdo it (sudo /sbin/route add -host n.n.n.n dev lo).

> Anyway, I'm happy to kick around further ideas about this any time, but
> it probably needs a chat rather than just e-mail.

I think we're tracking pretty well. I have a couple of very experienced
sysadmins in the team

Cheers

Jim

-- 
Jim Breen
Adjunct Snr Research Fellow, Japanese Studies Centre, Monash University