Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

does ask read robots.txt?

6:48 am on Jun 28, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2003
votes: 0

on ask.com faq I found this:

User-agent: Teoma
Disallow: /

, added it to my robots.txt (months ago), but they still read my website.

I really like the SE, but 0.006% of traffic sent from ask IMO doesn't justify them using my bandwidth over and over again.

I traced the spider-IP address and it traces back to some ask-address.

Any idea - apart from IP-Blocking and the other things meant for the bad boys?


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members