homepage Welcome to WebmasterWorld Guest from 54.161.236.92
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Ask - Teoma
Forum Library, Charter, Moderator: open

Ask - Teoma Forum

    
Ask Teoma bot crawling again
But...
Angonasec




msg:4234255
 12:28 am on Nov 24, 2010 (gmt 0)

We still specifically ALLOW the Ask Teoma bot for nostalgic reasons, and they haven't stopped crawling our site, but look at this weird UA, which got a speedy 403 of course...

204.236.147.79 - - [23/Nov/2010:15:30:56 -0500] "GET /robots.txt HTTP/1.0" 403 294 "-" "\"Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)\""

They visited using that format and IP.

Same day they used this format and got in:

66.235.124.15 - - [23/Nov/2010:03:30:47 -0500] "GET /robots.txt HTTP/1.1" 200 1221 "-" "Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)"


Deliberate, ham-fisted, or desperate?

 

bird




msg:4234422
 11:32 am on Nov 24, 2010 (gmt 0)

Is the first one really a Teoma IP? The UA rather looks like an impostor who doesn't know how to configure hist software correctly.

(edit: ) Actually, I just saw it's from amazonaws, another good reason to block that range.

Angonasec




msg:4234702
 10:52 pm on Nov 24, 2010 (gmt 0)

Thanks bird, I only checked the whois after you alerted me, and yes we block all amazonaws. This kind of abuse on their servers is dragging Amazon's reputation through the mud. They really ought to scrap amazonaws.

The second IP is genuine ask, so we still allow them :)

bird




msg:4234806
 6:59 am on Nov 25, 2010 (gmt 0)

Actually, amazonaws provides extremely useful services for a large number of purposes. They are quite unlikely to scrap that just because of a few disgruntled webmasters... ;)

That said, I'm probably going to block their ranges as well. If a legitimate and useful spider ever happens to operate from there, it's easy enough to make an exception.

Angonasec




msg:4235077
 11:51 pm on Nov 25, 2010 (gmt 0)

Q/
I'm probably going to block their ranges as well. If a legitimate and useful spider ever happens to operate from there, it's easy enough to make an exception.
/Q

Block all Amazon aws IP ranges, you won't regret it, nor will you ever be troubled to "make an exception".

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Ask - Teoma
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved