Site down during Deep Crawl

Will Google boot us out of the index?

[xbase234]

After devoting much time and effort to getting over 25,000 unique pages of our dynamic e-commerce catalog site spidered in Google, I fear we may lose these pages in the next update.

Our firewall crashed last week, and not having having a backup or supplier, we were down for 3 days. I ran log analysis reports that showed the Google bot had hit the site thousands of times a day just before the crash.

My question is this: Does anyone know the extent of damage this outage may have done to the number of pages indexed in Google?

Thanks in advance.

[korkus2000]

I don't think the deep crawler was out last week. It may have been freshbot getting everflux results. What was the crawlers IP?

[taxpod]

It's fine to be down after or before the deepcrawl. That was over weeks ago so you should have nothing to fear.

[xbase234]

I am showing 2,000 hits in one day last week from "Googlebot/2.1 ( [googlebot.com...] no IP given.

Would the Freshbot be hammering away at the site like this?

[EquityMind]

What PR is your site? That would determine how long between freshbot crawls. One of my sites had a server upgrade last week and was spidered by freshbot. We were out of the Google Database for about 12 hours before freshbot came back and put back all of our serps. This was for a PR7 site.

[Napoleon]

I doubt you have anything to worry about this time.

One other factor... if your site has PR of perhaps 5 or so and it is down for while during the deep crawl, Google apparently will re-visit. It seems to be a tolerant beast.

[webdevsf]

Speaking as someone who did something similar - you are lucky. the deep crawler stopped a few weeks ago, so this was probably the freshbot. The freshbot has only a minor impact on the serps compared to the deepbot.

If your firewall had been down during the deep crawl, there's a good chance that it would try to come back if it thought your site was down. I think it depends on your page rank.

But if your site was available but there was an error, ie, some programmers screwed up and put a 404 page for the deepbot, there is a good chance that you can get dropped out of the index for several months until deepbot decides to return.

So be very very careful. :)

[xbase234]

The site was completely down, "The page cannot be displayed", could not find page / dns, for 3 solid days. The bot got nothing.

Our PR is 6.

I will be greatly relieved if this was only the Freshbot and our pages will stay in the index. I just checked total pages, and we actually have 27,000 in the Google index.

[taxpod]

I've got a PR6 that routinely gets thousands of queries from Freshbot. I don't recall having seen deepcrawler over the past couple of weeks so I think you'll be OK.

As an aside, I had a really bad experience with a host when I was doing the $10-100 a month hosting packages. I don't know what the Google impact of that 48 hour outage was but I do know that I lost a good 30% of traffic as a result. Seems that the folks who were in the habit of visiting my site regularly to see updates became convinced that I had gone black for good. It took me months to get that traffic back. That's when I found a reputable dedicated serving company. Having your own server at a company which is rarely down for even an hour is worth the scratch you have to pay for it.

[stuntdubl]

You should be fine.

Googlebot is a tolerant beast.

I had a site down for over a month, and after having it back up I had regained many of my top serps (just through freshbot spidering). I'm not sure if it is back in the index (although I'm assuming so), because as we all know......someone is guarding the red button.

Bottom line....don't sweat the petty stuff.....
and don't pet the sweaty stuff.