joined:Mar 17, 2006
Hi all, messing up one's robots.txt can have really bad repercussions, so I would love to hear from people with more experience before completely destroying the site's SE traffic.
I have this rather unusual situation where I need to block the homepage, and only the homepage, of a site from indexing. I am moving a site penalized by Google (I'm going under assumption it's for bad links) and I don't want the bad links to follow (as in passing PR). Almost 100% of the bad links are to the homepage. There's a large number of good links to other pages, which I would like to salvage if at all possible.
So, is it even within the scope of the robots.txt protocol to block a page and not all the pages down the tree from it? Unless some trick exists, it appears to me that anything I do to the homepage in robots.txt will then be inherited by the rest of the site, so it will all vanish.
Can anyone comment on dealing with the homepage in robots.txt?