Forum Moderators: open

Message Too Old, No Replies

Yahoo ignoring robots.txt

and indexing pages that could get me banned?

         

internetheaven

9:37 am on Mar 16, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



My site has an insite search facility which is "protected" by robots.txt:

User-agent: *
Disallow: /search/

but today over 5,000 pages showed up in the Yahoo index and they are all search results pages for my site. e.g.

ht*p://www.mysite.com/search/search.php?Terms=widgets

I thought Yahoo was against search results pages? If so, will they ban my whole site when they see the 5,000 new pages even though it's not my fault they included them?