Re: [tlug] Blocking unknown and unclear bots

[Dave M G <dave@example.com>]

Curt,

> So why were people saying these bots were "bad"?
Short answer:

Crawling for emails or information to use for spam... maybe?

Longer answer:

What I do when I get an unrecognized user agent string is I look it up.

There are sites that list known good and bad bots. If the one I am
looking up is listed as bad, I block it in .htaccess, without really
worrying to much why that particular bot is bad. Some sites are kind of
vague or overly technical about the reasons.

Here, just by way of example, is a list of bad bots:

http://www.invision-graphics.com/robotstxt_badbots.html


d.