Welcome to WebmasterWorld Guest from 54.242.138.24

Forum Moderators: open

Message Too Old, No Replies

does ask read robots.txt?

     

the_nerd

6:48 am on Jun 28, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



on ask.com faq I found this:

User-agent: Teoma
Disallow: /

, added it to my robots.txt (months ago), but they still read my website.

I really like the SE, but 0.006% of traffic sent from ask IMO doesn't justify them using my bandwidth over and over again.

I traced the spider-IP address and it traces back to some ask-address.

Any idea - apart from IP-Blocking and the other things meant for the bad boys?

 

Featured Threads

Hot Threads This Week

Hot Threads This Month