How do I get Google to spider a site after using 301 re-directs

Having problems getting pages in the index

[Hinet]

We run a number of domains, and recently decided to use 301 re-directs to point to a single domain, as we thought the multiple domains may be causing mirrored site/duplicate content penalties.

The problem is that all of the pages we had in the index were under the old domains, and the new domain wasn't getting spidered. We hoped that after using 301 re-directs we would start to see the new domain getting spidered and old domain pages coming out of the index.

We have plenty of links pointing to the new domain, however after switching the domains over a month ago we still have none of the pages on the new domain in the index. This has meant that we have dropped rankings quite considerably.

We attempted to do the domain switching to ensure that only a single set of pages were in the index and to concentrate the effect of our links, content etc, but it seems to have backfired. Does anyone have any advice?

[DerekH]

The problem is that all of the pages we had in the index were under the old domains, and the new domain wasn't getting spidered. We hoped that after using 301 re-directs we would start to see the new domain getting spidered and old domain pages coming out of the index.

And the new domain still isn't getting spidered...

I think you have already answered your question - you need to discover why it wasn't (and hence why it isn't) being spidered.

It was broken, and now it's still broken - is that what you are saying?

But panic not...you also need to allow sufficient time for Google to respond. Some people are quoting days, if not hours, for Google to index the universe, whereas my sites get serviced (because of low PR) in the order of once every two months.

So panic or chill out - your choice.
If the former, what you have proved is that the problem you had hasn't gone away.
If the latter, you'll be really pleased soon. But not necessarily *very* soon.
DerekH

[mcavill]

Hi Hinet, I put up 301's on a site a few months ago - it was initally dropped, but eventually reindexed, and now ranks about the same in the SERP's as before. For my site, it took about 6 weeks from google picking up the 301's to the pages appearing in the index, it's now got the same links and toolbar PR as before the 301's.

I guess contacting your current link partners to update their links, and getting more links to the new domain would help ensure you get spidered ASAP.

[notsleepy]

You should create 301 redirects for each individual page on the old site to corresponding page on the new site.

The result is ZERO downtime. Worked perfect for me

[notsleepy]

Sorry to reply to my own post but heres an example:

OldSite/widget1.htm
301 Redirect to NewSite/widget1.htm

OldSite/widget2.htm
301 Redirect to NewSite/widget2.htm

OldSite/widget3.htm
301 Redirect to NewSite/widget3.htm

etc...

[Hinet]

Thanks for this, it's good to know others have had similar probs.

In terms of things being broken, I don't think there can be anything wrong with the site architecture, as the site is being spidered on other domains and other search engine's seem to have no problems with it.

The 301 re-directs work on every page on the site, and we have got a fair amount of links pointing to the main domain.

Is it possible that Google could have a spider block on the main domain as the same content has been found on other domains? If this is the case will 301 re-directs be enough to let them know that they can now start spidering the main domain?

I should probably just sit this one out and wait and see, however we are losing refferrals (and £'s) by the day and I'm getting increasing pressure from the powers that be, to do something about it. I have been waiting for something to happen for over 6 weeks now and the main domain is being visited reguarly but the only page in the index on this domain is the @*$$ing homepage.

[Marcia]

Do you have a site map for the new domain? It's recommended by Google to have one, and it might help things along a bit.

[Hinet]

A link to the sitmap is at the top of every page on the site.

We do use query strings to measure clicks on different parts of the page. Would this cause any issues? It never has before.

[Craig_F]

hinet, the same has been happening to me since last april. after much discussion with others I changed all my 301's to be set up like just like notsleepy mentions. it's only been a week or so for me, so I haven't seen a change yet, but I do think I will once Google comes back around.

[Hinet]

We have used a 301 that looks at where you were going on the old domain and re-directs to the same page on the new domain.

Will this method have the same effect as the way you are describing?

[chengfu]

To me it seems as if Google sometimes has problems changing the main domain for a website.
For a webshop I administer I have set up 301-redirects around christmas 2002 to redirect to the new main domain. Since then Google has refused to spider anything but the index-page.
Some weeks ago I decided to exclude the Googlebot from those redirects and only redirect "normal" webbrowsers. One day after this change the bot spidered the complete website (und the old domain) and updated all entriesin the index.
But it still refuses to spider the new domain...