| Welcome to WebmasterWorld Guest from 188.8.131.52 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
|does ask read robots.txt?|
on ask.com faq I found this:
, added it to my robots.txt (months ago), but they still read my website.
I really like the SE, but 0.006% of traffic sent from ask IMO doesn't justify them using my bandwidth over and over again.
I traced the spider-IP address and it traces back to some ask-address.
Any idea - apart from IP-Blocking and the other things meant for the bad boys?
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved