Yahoo have indexed and dredged up a listing from somewhere for an old site of mine (no snippets just the url and company name) where I have had the following robots.txt installed for a number of months. The site has NOT been available or listed anywhere for at least one year.
User-agent: * Disallow: /
Every page also contains robots exclusion meta tags
Is any one else seeing this or can advise any help on the robots.txt? It appears correct to me.
I have posted in the "Questions for Yahoo" forum a while ago without a response.
This is occurring for us to - a complete disregard to the robots.txt file. Yahoo! rep stated that YSlurp will obey the file, but I have weblogs that shows it just doesn't care. This is happening across multiple domains.