Forum Moderators: open
Next to the listings, these directories provide links to cached copies of the sites. These cached copies are clearly spiderable since many are listed in Google and have PR.
I am very concerned about the issue of duplicate content, which could lead to sites being penalized by certain search engines.
I discovered this problem because they recently added a listing to one of my sites. I did not request this listing. I don't mind the link, but I really don't like the fact that there are now two copies of my index page in Google. Don't you think that something should be done to avoid the potential harm to innocent web sites that this type of directory could do?
They don't
Disallow: /common/cache/
like they should - naughty.
