Forum Moderators: goodroi
I can then fix a few problems that have been bothering me and implement a htm rewriter on my pages.
The site currently has no google PR at the moment.
If I then reallow robots.txt after a week will everything be OK
Any thoughts please
DaveN
What would yahoo do - would they just remove all the pages and then kind of treat it as a new site, reindex and hopefully quickly rerank?
Would there be a sandbox effect?
Would love to hear of any webmasters who have done this to hear your views.
cheers
[help.yahoo.com...]
DaveN
Would the site go back into the sandbox? Probably not. Google would still be tracking links to you, would still be crawling all your pages and will have your entire site on the back burner. Even if it didn't - the site isn't a new site (BTW, what sandbox? ;)) so unlikely to suffer the so called sandbox effect. Disclaimer: I've never done this myself.
I suspect the supplementals will be on a back burner as well. If getting rid of supplementals was a simple matter of listing them in a robots.txt we wouldn't have all the complaints here about supplementals that just keep popping back after six months. Ask people like steveb. There is apparently no known way you can get Google to completely remove a page of yours they've got in supplemental.
That the site has no PR is not something you know for certain. Despite the robots.txt Google will crawl your pages, will follow links, and will likely award PR from you to other sites you've linked to.
Maybe it is a bad bot :). I could be completely wrong on this, of course. It may be a technical issue that causes Google to crawl. Perhaps they follow a link from somewhere and read your file (but do it without checking for the robots.txt everytime they visit). And, it may even be that they've only committed to not listing banned pages. From DaveN's link above Yahoo is very clear that they will not crawl disallowed pages.
[edited by: oddsod at 3:44 pm (utc) on Nov. 23, 2005]
Going to try this method i think
DaveN
Blocking spiders in robots.txt leaves you vulnerable to competitors filing yourdomain.com/ in Google's removal tool. If they do Google's only security check is verifying your robots for confirmation. They'll find the block there and yourdomain.com is nuked for 180 days. You can change your robots file and do pretty much anything else it won't make a difference.... you'll be down for six months.
If you believe it's worth a shot to ban bots to get rid of supplementals - why ban all bots? Only Google has this problem and you can selectively block the googlebot.
I want to implenent a URL rewriter and change all the pages to metaname.htm
I am thinking this will lead to duplicate content and other problems.
So I am thinking that a week of banning via robots.txt and then implenent the rewriter and see what happens.
If you have any more comments then keep them coming.
cheers
