Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: open

Message Too Old, No Replies

Ask Teoma bot crawling again

But...

     
12:28 am on Nov 24, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


We still specifically ALLOW the Ask Teoma bot for nostalgic reasons, and they haven't stopped crawling our site, but look at this weird UA, which got a speedy 403 of course...

204.236.147.79 - - [23/Nov/2010:15:30:56 -0500] "GET /robots.txt HTTP/1.0" 403 294 "-" "\"Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)\""

They visited using that format and IP.

Same day they used this format and got in:

66.235.124.15 - - [23/Nov/2010:03:30:47 -0500] "GET /robots.txt HTTP/1.1" 200 1221 "-" "Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)"


Deliberate, ham-fisted, or desperate?
11:32 am on Nov 24, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 10, 2001
posts:1550
votes: 10


Is the first one really a Teoma IP? The UA rather looks like an impostor who doesn't know how to configure hist software correctly.

(edit: ) Actually, I just saw it's from amazonaws, another good reason to block that range.
10:52 pm on Nov 24, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


Thanks bird, I only checked the whois after you alerted me, and yes we block all amazonaws. This kind of abuse on their servers is dragging Amazon's reputation through the mud. They really ought to scrap amazonaws.

The second IP is genuine ask, so we still allow them :)
6:59 am on Nov 25, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 10, 2001
posts:1550
votes: 10


Actually, amazonaws provides extremely useful services for a large number of purposes. They are quite unlikely to scrap that just because of a few disgruntled webmasters... ;)

That said, I'm probably going to block their ranges as well. If a legitimate and useful spider ever happens to operate from there, it's easy enough to make an exception.
11:51 pm on Nov 25, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


Q/
I'm probably going to block their ranges as well. If a legitimate and useful spider ever happens to operate from there, it's easy enough to make an exception.
/Q

Block all Amazon aws IP ranges, you won't regret it, nor will you ever be troubled to "make an exception".