Wy don`t Crawl Google today?

[msebian]

Dear Guys,
first I´m sorry for my terrible English:).
I have see,that google crawl last month everydays, and this month nothig.
Has google last month only make a test?
Thanx
Mike

[percentages]

Hi msebian,

Googlebot will crawl most days this month also. At the moment it is finishing up the monthly update. Once that is done the Google "Freshbot" will be out and crawling....sooner the better IMO;)

The Google Freshbot is still a new kid on the block and she doesn't work everyday, especially during the "dance" period. However, we all love her and like you I can't wait to see her again:)

[msebian]

The last Update was a Horror Update for me, 50% less traffic to the forlast update.
The PR is going to Nirvana from 6 to 3 and a must site was nothing in the index.
I hope google is comming everyday,that crawl my sites, for mre traffic:(
Thx a lot
Mike

[xlcus]

I have see,that google crawl last month everydays, and this month nothig

Google has started this month's crawl. I had a few visits yesterday and a lot more today from a deep-crawl bot.

I'm sure one of the bots will get round to you soon.

[Yidaki]

Google's deepcrawling most of my sites since yesterday.

[iamjoe]

I believe google just started to deep crawl my site.

crawl13.googlebot.com - - [01/Jan/2003:08:35:28 -0500] "GET /robots.txt HTTP/1.0" 200 48 "-" "Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)"
crawl13.googlebot.com - - [01/Jan/2003:08:35:28 -0500] "GET /images/index_r1_c01.jpg HTTP/1.0" 200 10732 "-" "Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)"
crawl16.googlebot.com - - [01/Jan/2003:14:42:54 -0500] "GET /images/index_r1_c08.jpg HTTP/1.0" 200 4981 "-" "Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)"
crawl6.googlebot.com - - [03/Jan/2003:20:25:00 -0500] "GET /robots.txt HTTP/1.0" 200 25 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
crawl6.googlebot.com - - [03/Jan/2003:20:25:00 -0500] "GET / HTTP/1.0" 200 6579 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
crawl7.googlebot.com - - [05/Jan/2003:07:53:36 -0500] "GET /robots.txt HTTP/1.0" 404 275 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
crawl7.googlebot.com - - [05/Jan/2003:07:53:38 -0500] "GET /dialup.phtml HTTP/1.0" 200 6896 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Seems like it took google a bit of time and pulling the / page a few times before it started to go deeper than the / path.

The one other thing i had done from when it last visited me on the 3rd was delete my robots.txt file

All reference i see to robots.txt is to exclude content and not actually add it in. I had found one reference on how to actual "ALLOW" the bot via the robots.txt.

What i find interesting is this:
crawl7.googlebot.com - - [05/Jan/2003:07:53:36 -0500] "GET /robots.txt HTTP/1.0" 404 275 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
crawl7.googlebot.com - - [05/Jan/2003:07:53:38 -0500] "GET /dialup.phtml HTTP/1.0" 200 6896 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Notice the 404 on the robots.txt and how it downloaded another page? Didnt do that the other 2 times when i served the robots.txt with the allow directive.

[nutsandbolts]

Give it chance to visit your site - the Web is a big place ;) I would only start to worry if The Fonz hasn't crawled you by the middle/end of this week.

[silinx]

The deep crawl has been hitting me all day yesterday and today, so far. It'll come around.

;)