Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt for mirror sites


jdMorgan - 10:17 pm on Aug 6, 2002 (gmt 0)


ScottM,

Yes, put the disallow only on the mirror sites, not the main site.

If Google (for example) indexes a page on your mirror site, and (for some reason*) assigns it a
higher PR than the page on your main site, it might drop the main site page and list the mirror page
instead - probably not what you want. Some here say that if it finds too much duplicated content or
too much cross-linking then it will give you a PR0. Some say that won't happen, some say it will.

I'm not sure, but be careful.

Jim

* This could happen if a competitor intentionally linked his very-high-PR page to your mirror page,
as pageoneresults implied.


Thread source:: http://www.webmasterworld.com/robots_txt/144.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com