Forum Moderators: open
A very old site which is subject to robots.txt
User-agent: *
Disallow: /
and robots exclusion tags in every page has just appeared in Y's new search results. This URL is not listed anywhere within the Y's directory or any other directory/search engine.
As far as I can tell the index.html page only for the correct site, which is listed in Y's directory, has vanished from search results completely and been replaced by the url which is subject to the robots.txt There are no redirects involved with either url.
Y's bot appears not to be obeying the robots.txt.
Any help appreciated?
Further info available where required.
Regards to all
Ray
EW
Should have been posted in
Questions for Yahoo!
Round them up and post them here.
[webmasterworld.com...]
Please move
Thanks
EW