Welcome to WebmasterWorld Guest from 54.159.12.17

Forum Moderators: open

Message Too Old, No Replies

Ask Teoma bot crawling again

But...

     

Angonasec

12:28 am on Nov 24, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We still specifically ALLOW the Ask Teoma bot for nostalgic reasons, and they haven't stopped crawling our site, but look at this weird UA, which got a speedy 403 of course...

204.236.147.79 - - [23/Nov/2010:15:30:56 -0500] "GET /robots.txt HTTP/1.0" 403 294 "-" "\"Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)\""

They visited using that format and IP.

Same day they used this format and got in:

66.235.124.15 - - [23/Nov/2010:03:30:47 -0500] "GET /robots.txt HTTP/1.1" 200 1221 "-" "Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)"


Deliberate, ham-fisted, or desperate?

bird

11:32 am on Nov 24, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Is the first one really a Teoma IP? The UA rather looks like an impostor who doesn't know how to configure hist software correctly.

(edit: ) Actually, I just saw it's from amazonaws, another good reason to block that range.

Angonasec

10:52 pm on Nov 24, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks bird, I only checked the whois after you alerted me, and yes we block all amazonaws. This kind of abuse on their servers is dragging Amazon's reputation through the mud. They really ought to scrap amazonaws.

The second IP is genuine ask, so we still allow them :)

bird

6:59 am on Nov 25, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Actually, amazonaws provides extremely useful services for a large number of purposes. They are quite unlikely to scrap that just because of a few disgruntled webmasters... ;)

That said, I'm probably going to block their ranges as well. If a legitimate and useful spider ever happens to operate from there, it's easy enough to make an exception.

Angonasec

11:51 pm on Nov 25, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Q/
I'm probably going to block their ranges as well. If a legitimate and useful spider ever happens to operate from there, it's easy enough to make an exception.
/Q

Block all Amazon aws IP ranges, you won't regret it, nor will you ever be troubled to "make an exception".
 

Featured Threads

Hot Threads This Week

Hot Threads This Month