| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Subscribe and Support WebmasterWorld|
|does ask read robots.txt?|
| 6:48 am on Jun 28, 2006 (gmt 0)|
on ask.com faq I found this:
, added it to my robots.txt (months ago), but they still read my website.
I really like the SE, but 0.006% of traffic sent from ask IMO doesn't justify them using my bandwidth over and over again.
I traced the spider-IP address and it traces back to some ask-address.
Any idea - apart from IP-Blocking and the other things meant for the bad boys?
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved