Ask Teoma bot crawling again

But...

[Angonasec]

We still specifically ALLOW the Ask Teoma bot for nostalgic reasons, and they haven't stopped crawling our site, but look at this weird UA, which got a speedy 403 of course...

204.236.147.79 - - [23/Nov/2010:15:30:56 -0500] "GET /robots.txt HTTP/1.0" 403 294 "-" "\"Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)\""

They visited using that format and IP.

Same day they used this format and got in:

66.235.124.15 - - [23/Nov/2010:03:30:47 -0500] "GET /robots.txt HTTP/1.1" 200 1221 "-" "Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)"


Deliberate, ham-fisted, or desperate?

[bird]

Is the first one really a Teoma IP? The UA rather looks like an impostor who doesn't know how to configure hist software correctly.

(edit: ) Actually, I just saw it's from amazonaws, another good reason to block that range.

[Angonasec]

Thanks bird, I only checked the whois after you alerted me, and yes we block all amazonaws. This kind of abuse on their servers is dragging Amazon's reputation through the mud. They really ought to scrap amazonaws.

The second IP is genuine ask, so we still allow them :)

[bird]

Actually, amazonaws provides extremely useful services for a large number of purposes. They are quite unlikely to scrap that just because of a few disgruntled webmasters... ;)

That said, I'm probably going to block their ranges as well. If a legitimate and useful spider ever happens to operate from there, it's easy enough to make an exception.

[Angonasec]

Q/
I'm probably going to block their ranges as well. If a legitimate and useful spider ever happens to operate from there, it's easy enough to make an exception.
/Q

Block all Amazon aws IP ranges, you won't regret it, nor will you ever be troubled to "make an exception".