Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Deep Crawl/Site Down

7:27 am on Jan 4, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 11, 2002
votes: 0

Hi Guys,

My site was down for 8 hours today (I've since purchased some back-up "insurance" in case this happens again).

Here's the deal:

The deep crawler (216.239....) hit my robots.txt and my index page about 10 mins. before I crashed...

Will I still be in the index next update? Is it going to drop any of my other pages that didn't get hit today?

Freshbot crawls me every day, but I know all my interior pages missed the deep crawl (at least for today) and only my main page got hit.

Any opinions?



7:35 am on Jan 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Oct 1, 2002
votes: 2


To me Google seems fairly tollerant on pages that are down at a particular instance. It may not get the updates, but it doesn't seem to drop pages simply because it couldn't get to them on the first attempt.

Reliable hosting is however invaluable. Suffer this fait a couple of times in a row and you maybe in trouble.

However, for a one time problem I don't think Google will care :)

7:43 am on Jan 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 5, 2001
votes: 0

It's also not completely certain that you've missed the deep crawl, in spite of having seen that robots.txt/index.html hit. For example, the site I'm watching most closely this update got visited with that same request last night sometime before 10 pm (when the logs rolled over), but the first deep crawl hits began just a few hours ago -- more than 24 hours later.

While there's no way to predict what might happen, I wouldn't panic yet.

7:45 am on Jan 4, 2003 (gmt 0)

Junior Member

joined:Oct 19, 2002
votes: 0

I am going here on my own past experiences -
First has googlebot or freshbot hit your site after it came back up? I have often seen googlebot come back within 24 hours to re-check a site durring the deep crawl. (Iím assuming that you have some good links pointing to your site.)

If you donít see googlebot or the newer freshbot after a week or so you might possibly be out of the next index.