homepage Welcome to WebmasterWorld Guest from 54.226.213.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Ask - Teoma
Forum Library, Charter, Moderator: open

Ask - Teoma Forum

    
does ask read robots.txt?
the_nerd




msg:33202
 6:48 am on Jun 28, 2006 (gmt 0)

on ask.com faq I found this:

User-agent: Teoma
Disallow: /

, added it to my robots.txt (months ago), but they still read my website.

I really like the SE, but 0.006% of traffic sent from ask IMO doesn't justify them using my bandwidth over and over again.

I traced the spider-IP address and it traces back to some ask-address.

Any idea - apart from IP-Blocking and the other things meant for the bad boys?

 

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Ask - Teoma
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved