Forum Moderators: open
We'd like to get some links for a new site, but the site is not complete and we don't want to let Googlebot in yet. (site may be viewed as dupe content if released before complete)
1) Will the disallow delay Googlebot's return? Or does Googlebot show up as frequently, even if each time it does, the robots.txt says "disallow all"?
2) Will there be any long-term harm of this method (getting links while disallowing Googlebot)?
I don’t know your exact situation, but unless there is something very wrong with the pages you are building. I would let the bot see the new changes. Remember when the bot reads something new every time it comes to your website, then it will come back more frequently. I don’t know what you mean about dupe content, but I would let the bot in just for the PR to get an early start. Keep in mind that it can take several months to get the PR to finally show up on each page.
There’s my two cents for what it’s worth.
CygnusX1
A couple of sites just recently have been excluded from the index, and while there may be other factors involved, it appears that duplication on two different tld's may just be the reason.
Unless you can cordon off the portion of the site that's a duplication and keep it completely separate (i.e. password protected) and put up some unique accesssible pages to start getting links and PR, it would be a whole lot safer to wait until the issue of duplication isn't at all a potential for a problem.
Even with Googlebot being disallowed, if pages on the site are found through links they can be included and appear in the index with URL only, no title and no description.
There are suspicions that duplicate content isn't as light a matter now as it used to be.
There are suspicions that duplicate content isn't as light a matter now as it used to be.
I've experienced this problem on a chain of sites that use a lot of duplicate content, I have to agree with this statement. I've tried telling my client this but the ease of maintainance on his and my end have made it too hard to create unique sites, not worth it, as our serps plummet, we used to have 4 sites in top 10 for one keyword phrase, then they all dropped out at once.
Does anybody have a clear idea, tested that is, not a guess, on how much difference there needs to be per page before they are again considered unique? Eg, title different, headers different, some words different per paragraph?
Remaining question: will a "disallow all" slow the bot from coming back? For example, let's say I have a link from a PR7 site, and I disallow all bots. Normally, Googlebot would come pretty frequently. If I disallow all, will it be a while before G bothers coming back? Or does G NOT get "discouraged" if each time it comes it is told to "go away"....
Does the fact that it is told to "go away" slow down its coming back?
