Re: [tlug] Blocking unknown and unclear bots

[Darren Cook <darren@example.com>]

> Here, just by way of example, is a list of bad bots:
> 
> http://www.invision-graphics.com/robotstxt_badbots.html

It is fascinating that spammers would identify themselves with a user
agent of "EmailSiphon" :-)

But these two at least are general purpose:
  Microsoft URL Control - 5.01.4511
  Microsoft URL Control - 6.00.8169

I.e. they might be used for a valid purpose. (Anti-Microsoft comments to
your own thread please.)

There are two main approaches on this discussion:
  http://www.webmasterworld.com/forum11/1005.htm

1. Block by user-agent;
2. Secure your forms (by only allowing POST, using a captcha, only allow
it to send to one account, etc.) instead of leaving them open for anyone
to use to send email to anyone.

The latter is an example of what Curt meant by it is better to filter by
behaviour.

As for email address harvesting: have you received even just one spam
message the past week? If so it is too late, the horse has already bolted.

For denial-of-service attacks, watching for lots of activity from the
same IP address and slowing it down, is a better approach. (A quick
Google shows mod_security can do this for Apache, but I've no personal
experience.)

Darren