homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Is the crawl done?
No hits from Deep Crawler in 4 hours.

 12:48 pm on Mar 18, 2003 (gmt 0)

Should I be worried? GoogleBot (deep) has not hit any of my 5 sites in about 4 hours. My major concern is Google has gotten about half the number of pages so far this crawl as last crawl.

It started very slowly this month, and didn't start getting serious until a few days into the crawl.

Am I alone in GoogleBot being gone? Is it done?



Alternative Future

 12:54 pm on Mar 18, 2003 (gmt 0)

Nope still kicking around here. Been a little slower than in the past though!



 12:55 pm on Mar 18, 2003 (gmt 0)

I have the exact same experience!


 3:19 pm on Mar 18, 2003 (gmt 0)

After a couple of intense days during which the Googlebots crawled about 55000 pages (which is considerably less than ususal), things went calm about 4am this morning. No sign of the Deep Crawlers since that time. So if the Deep Crawl is over, it's been a much less extensive one for me than the last couple of months...


 4:20 pm on Mar 18, 2003 (gmt 0)


I have been experiencing the same. Since, the major deep crawl on March 10th, my sites have not been deep crawled even once. The freshbot is also slow.


 5:11 pm on Mar 18, 2003 (gmt 0)

I think I scared him away this morning!


 5:14 pm on Mar 18, 2003 (gmt 0)

I saw about 25000 hits a day for a week, but today only the freshbot. Funny thing is - I only have 60000 some odd pages so it would seem deepbot crawed my entire site more than once (although I have not confirmed that...)


 5:53 pm on Mar 18, 2003 (gmt 0)

Only slow freshbot since 3AM here too. Is he bringing up the rear for Deepbot? Get back to work Deepbot, you're not done.


 6:53 pm on Mar 18, 2003 (gmt 0)

Deepbot is back!


 8:00 pm on Mar 18, 2003 (gmt 0)

Yes, the Deep Crawlers have returned to my site as well. I'd be interested to know whether the Deep Crawl takes a full-on intermission or whether it simply moves on to other sites for a time. In any case, I'm pleased to have it back in the house...


 8:00 pm on Mar 18, 2003 (gmt 0)

coukd it be that you are a bit paranoid? ;-)

itīs the end of the spider cycle, so only tomorrow and maybe the day after tomorrow the deepbot is on the way. then itīs over for a few weeks.


 9:11 pm on Mar 18, 2003 (gmt 0)

Back for me as well. (I started this thread.)

Anyone know this to have happened before? It is obviously more than just me, as other people saw it stop, and come back more or less on all sites at once.

Hopefully it still has time to get the other half of the pages it usuallty gets.



 10:03 pm on Mar 18, 2003 (gmt 0)

Still here...


 4:01 am on Mar 19, 2003 (gmt 0)

Well wouldn't depend on how the google database is designed? and how the bots search - is it by domain name or ip address of the server, or the number of pages he wants to look at?

On my server I've seen an almost perfect bell curve starting 3/11 peaking 3/15 and decling down today 3/18


 4:55 am on Mar 19, 2003 (gmt 0)

IMHO DeepCrawl is not done just yet. Still some pages being spidered. It did take a break though which was noticeable.

And just to scare people. I have one site that was deep crawled back in December and has never been deep crawled again. Don't ask me why. The PRs are still fine. Freshbot appears from time to time, but deepbot is not there.

Sometimes we webmasters panic a little too readily.


 7:09 am on Mar 19, 2003 (gmt 0)

normally the deepcrawl is separated into to sections with a little break between them. (break doesnīt mean, that it doesnīt come at all - it only takes a very few pages during this time) the first part takes about 5 to 6 days, the second one 3 to 5.


 11:39 am on Mar 19, 2003 (gmt 0)

Googlebot is crawling my server since march 13th and it is still there. Donīt stop babyyy :)


 12:31 pm on Mar 19, 2003 (gmt 0)

DeepBot still here.

Btw...how long a deepcrawl usually go on?




 12:51 am on Mar 20, 2003 (gmt 0)

10 days


 2:22 am on Mar 20, 2003 (gmt 0)

i dont get why the deepcrawl doesnt hit all of the pages its supposed to hit. any simple explainations or ways to get it, maybe its beyond our control?


 4:58 pm on Mar 20, 2003 (gmt 0)

Same there. Deepcrawl was constantly grabbing pages then it abruptly stop and does not come back.

Something wrong. Something to do with the war or what?



 5:42 pm on Mar 20, 2003 (gmt 0)

Since I first posted to this thread a couple of days ago, the Deep Cralers have been coming and going with no real pattern. There have been hours where I served them more than 1000 pages and others where it's been less than 50. It's not idea but so long as it continues I'll happily endure it...

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved