massive crawl google

from 25 to 75 crawls a day

[krbulldog]

normal my sites are visited about 25 times a day by a googlebot, today they have visited about 75 times bevore 12:00. I'm going to hit over 100 today?

How is this possible?

[xlcus]

Perhaps you're being crawled by both the DeepCrawlBot and the FreshBot at the same time.

[krbulldog]

could by, how can I reconize them?

they're all crawl(1 to 9).googlebot.com
and from Googlebot/2.1 (+http://www.googlebot.com/bot.html)

[Grumpus]

About every third or fourth update, Google seems to do what I call "The Hail Mary Crawl" (i.e. They Go Deep). This seems to be one of those months as it's the 10th and there's still no sign of slowing down.

If it's not hurting you, then let it crawl. You should get some good representation next month.

G.

[Rugles]

krbulldog

The google deep crawing comes from the 216 IP range. The Fresh bot comes from the 64 IP range.

I would suggest that you welcome Mr. Googlebot with open arms and jump for joy when he is aggressively crawling your site. If you did not see him you would miss him.

[krbulldog]

of course i'm happy with it. :) he visited my site now more than 100 times today :)

[HuhuFruFru]

?

t thought the deepcrawl is only once a month for a few days after the update has been finished?
did i miss sth?

[HuhuFruFru]

but i can see the deepcrawler on my site too in my log-file

[korkus2000]

Every couple of months there is a super crawl. You will see a lot more of GoogleBot that month.

[HuhuFruFru]

super crawl?

i never heard of that. why a deep crawl at the beginning of the month and then another one now? what's the difference?

[krbulldog]

I have everyday googlebots in my log. He's gone mad today.. Now even over the 150.

I submit sub-sites everyday and it seems to work very well now :)

But i'm still on a pr 5 :(

[mfishy]

Since Google is still deep crawling, does that mean that sites getting crawled now will be included in the next update?

I just published a website and it was first crawled on the 8th. I was assuming it would be too late for next month, but now I'm not sure.

[Rugles]

What ever is being deep-crawled in the last week or so will show up in the index at the end of the month.

[ExtremeExports]

I was wondering how I can tell if the googlebot comes to my site? I've been noticing here in the forum people writing that the google bot or fresh bot came this morning or this evening. How can you tell? I have a webalizer on my cpanel and I also have the analog. but, i still cant tell when and if the google bot or fresh bot have been visiting my site.

[VictorE]

Since Google is still deep crawling, does that mean that sites getting crawled now will be included in the next update?

I just published a website and it was first crawled on the 8th. I was assuming it would be too late for next month, but now I'm not sure.

The "DeepCrawler" paid me a visit today, too. However, he only grabbed my homepage. I checked my logs for December, and he did the exact same thing exactly one month ago (Dec 10). My site has already been fully DeepCrawled earlier this month. It was already fully DeepCrawled when he showed up for my home page on December 10, too.

I don't think that I would expect that since the DeepCrawler is out and about he will pick you up for the Jan update. However, best of luck!

Vic

[zeus]

You have the same problem like me I just have to many visits everyday so the googlebot is not showing up because of to little hits from the google bot.

zeus

[gsmitchell]

extremeexports

Here is a webmasterworld link concerning Google
[webmasterworld.com...]

also here is info on the Googlebot
Googlebot/2.1d (+http://www.googlebot.com/bot.html)

If Google has been there it should be in your log files

[Hollywood]

FYI - I have a hit today or yesterday

01:02:10 PM crawl1.googlebot.com (216.239.46.27)

Still trying to figure out what this all means, when people have the chance here please feel free to go an and explain all you know about this process.

Ciao - Hollywood

[BigDave]

My site was completely indexed by the 7th, with nothing since then. I suppose that it might be google trying to work it's way towards that 10 billion page mark.

I certainly don't mind if they do, a lot of those borderline pages link to me. Their votes may not count for a lot, but they still count for something.

[Hollywood]

Anyone have any ideas who these hits are from?

10 Hits this month - drone11.sv.av.com

[Rugles]

Hollywood

That other bot is from Alta Vista.

VictorE
The google deep crawler was out in the last 24 hours. Grabbed many of our pages. I have seen this behavior before and I do expect to see these pages in the update at the end of this month

[xlcus]

I just have to many visits everyday so the googlebot is not showing up because of too little hits

If you're using Apache to run your own website and want to easily see visits from the GoogleBot, you could try something similar to the following in your config...

<VirtualHost 192.168.0.1> 
 ServerName mysite.com 
 DocumentRoot /sites/mysite.com/htdocs 
 ErrorLog /sites/mysite.com/error_log 
 SetEnvIfNoCase User-Agent Googlebot isrobot=true 
 CustomLog /sites/mysite.com/access_log combined env=!isrobot 
 CustomLog /sites/mysite.com/robots_log combined env=isrobot 
</VirtualHost>

This puts any accesses from the GoogleBot in a separate log file.

[charpress]

I get visited by Googlebot pretty much on a daily basis.

I pay more attention to how much Googlebot pulls compared to other crawlers and it is typically about 10 time the average of the next most active crawler. This is usually about 1 meg per day, but can be twice that. I know this will vary, obviously, in accordance with the size of the site that is crawled, but what do the rest of you see as far as bytes pulled by Googlebot in a typical day?