Re: [tlug] Blocking unknown and unclear bots

[Dave M G <dave@example.com>]

Curt, Darren,

Thank you for replying.

Yeah, I know that my system isn't perfect and won't protect against bots
that fake their user agent string.

However, my time resources are limited, so unless some bot does
something truly egregious, I can't be monitoring all their activity.

My user agent string sniffing basically started out as just a way of
sorting out mobile browsers from desktops. The logging was mainly done
to check for new browsers and mobile devices, so that I could keep up to
date on what kind of devices and capabilities were looking at the site.

Only one of my sites redirects based on mobile versus desktop, but it's
a capability I want to keep as an option.

I figured that while I was at it, if I could block some obviously bad
bots, then why not. But I'm reliant on word-of-mouth spread on the web
for what counts as "bad".

I'm assuming anyone who is up to something truly malicious can fake
whatever they want, which raises the question:

Why would any malicious bot ever identify itself as anything other than
a trusted bot from Google or Mozilla or something of that nature? Why do
they give themselves names that can be found and filtered?

Also, the other question I now wonder is, what is the worst thing a bot
or webcrawler could do? DoS attack maybe? Seems like other than the
unlikely event that anyone cares about my extremely low traffic sites
enough to launch a DoS attack, the only thing that could happen is them
searching and indexing my pages.

Is there something else I should be worried about?

-- 
Dave M G