homepage Welcome to WebmasterWorld Guest from 54.204.90.135
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Deep Crawl/Site Down
jen24815




msg:166408
 7:27 am on Jan 4, 2003 (gmt 0)

Hi Guys,

My site was down for 8 hours today (I've since purchased some back-up "insurance" in case this happens again).

Here's the deal:

The deep crawler (216.239....) hit my robots.txt and my index page about 10 mins. before I crashed...

Will I still be in the index next update? Is it going to drop any of my other pages that didn't get hit today?

Freshbot crawls me every day, but I know all my interior pages missed the deep crawl (at least for today) and only my main page got hit.

Any opinions?

Thanks!

Jenny

 

percentages




msg:166409
 7:35 am on Jan 4, 2003 (gmt 0)

Jenny,

To me Google seems fairly tollerant on pages that are down at a particular instance. It may not get the updates, but it doesn't seem to drop pages simply because it couldn't get to them on the first attempt.

Reliable hosting is however invaluable. Suffer this fait a couple of times in a row and you maybe in trouble.

However, for a one time problem I don't think Google will care :)

JayC




msg:166410
 7:43 am on Jan 4, 2003 (gmt 0)

It's also not completely certain that you've missed the deep crawl, in spite of having seen that robots.txt/index.html hit. For example, the site I'm watching most closely this update got visited with that same request last night sometime before 10 pm (when the logs rolled over), but the first deep crawl hits began just a few hours ago -- more than 24 hours later.

While there's no way to predict what might happen, I wouldn't panic yet.

Lots0




msg:166411
 7:45 am on Jan 4, 2003 (gmt 0)

I am going here on my own past experiences -
First has googlebot or freshbot hit your site after it came back up? I have often seen googlebot come back within 24 hours to re-check a site durring the deep crawl. (Iím assuming that you have some good links pointing to your site.)

If you donít see googlebot or the newer freshbot after a week or so you might possibly be out of the next index.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved