Forum Moderators: open

Message Too Old, No Replies

robots.txt

Y not obeying robots.txt

         

EarWig

8:47 am on Mar 20, 2004 (gmt 0)

10+ Year Member



Thanks to WebGuerrilla and Yahoo for this thread and the response from Y employees - Excellent to be able to get some feedback and help.

A very old site which is subject to robots.txt
User-agent: *
Disallow: /
and robots exclusion tags in every page has just appeared in Y's new search results. This URL is not listed anywhere within the Y's directory or any other directory/search engine.

As far as I can tell the index.html page only for the correct site, which is listed in Y's directory, has vanished from search results completely and been replaced by the url which is subject to the robots.txt There are no redirects involved with either url.

Y's bot appears not to be obeying the robots.txt.
Any help appreciated?
Further info available where required.
Regards to all
Ray
EW

EarWig

1:57 pm on Mar 22, 2004 (gmt 0)

10+ Year Member



Mods - Apologies

Should have been posted in
Questions for Yahoo!
Round them up and post them here.
[webmasterworld.com...]
Please move
Thanks
EW