Deep Crawl

Googlebot on the prowl!

[hoteimports]

Googlebot is on the prowl! Just had a very, very dep crawl. Got all 18,000 pages!

[layer8]

I hope its a fashion thing, its been crawling every day but its too shy to come in so I'm waiting for a nice deep crawl.

I'll let you know if it happens

[larryhatch]

Very same thing here on a smaller scale.
I only have 150 unique pages or so, looks like G hit them all, or maybe a few of them twice. - Larry

[Livenomadic]

Oh man that would be sweet is I got a deep crawl, I just doubled my pages yesterday.

[Spine]

They are doing something. Right now they are trying to crawl a bunch of long deleted pages on one of my sites.

It seems every page they are looking for is an old deleted one, and none of them are the ones acutally on my site at this moment.

Weird.

[GerBot]

I got hit on the 23rd (all pages) which would be great if I had not uploaded my new pages on the 24th :(

[bloke in a box]

I got hit yesterday for 3k pages which is virtually the entire site.

[shabir]

PR went grey for all sites for me in India..

[inbound]

First deep crawl for one of my new sites today (UK), it took them long enough but I'm pleased.

Googlebot back from vacation and feeling ravenous.

[Kukenan]

Oh my God this is beautiful!

For the first time EVER I see googlebot following 301 redirects! it hits the old page and immediately after it gets the new page "200"

I have never seen it do that before!

I am going to cry! :-)

[BillyS]

My site was deep crawled starting last night through the early morning. Took in the entire site - around 600 pages currently.

If this follows the same pattern as the last couple of months, this means we might be several days off (Wednesday perhaps) away from a change in SERPs.

[Clark]

They're trying to catch all those new nofollow rel attributes. But they will have to do fairly frequent crawling because it will take time until it's implemented. So enjoy the next few months.

[webdude]

Yup. I am seeing the same thing, albiet very slow. One of my sites is in the process of being crawled right now. It started on Saturday and is still going on. It appears that googlebot is hitting every one of my pages. On average it looks to be one page every 7 minutes. It will come and slurp 2 pages in about 14 minutes or 1 every 7. Not like the old days when it would hit the whole site in less then an hour.

[Critter]

I'm seeing the exact opposite. Have 1/20th the traffic from Googlebot as usual. Very quiet (I'm scared).

:)

[Critter]

Oops, spoke too soon.

It's baaaack.

[speedshopping]

I am currently being crawled but googlebot is spending on average about 10 minutes on each page before moving to another one? Is this normal - my site is currently at a URL only status?

[johnnie]

Weird, I'm being crawled in short bursts. GB hops in, crawls some 5 pages and leaves. One hour later it comes back and crawls approx. 5 others.

Are they trying to catch that new tag?

[Nikke]

145 visits by Google's bots yesterday, and 139 of those by the one identifying itself as
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

This far, today, 44 out of 56 googlebot visits are by the Mozilla/5.0 type.

For the record: I have implemented the nofollow link attribute on most of my affiliate links.

Edited some spelling

[robster124]

My new site is getting a deep crawl as we speak...

[Nikke]

I went back and checked the logs for Sunday, and found that that was when the deep crawl of my site occured.
445 visits, and just 39 of them by the Mozilla/5.0 type of bot.

[pmkpmk]

Where is the significance in thos Mozilla-bot-thing?

[johnnie]

I don't know... Maybe the moz-5.0 version accepts newer standards?

[webdude]

Mozilla/5.0 on all of mine. Still in the process of a deep crawl. Started Saturday and still not done. 25 more pages were added in the SERPs but they just show the the URL currently. Yet to pick up the titles of the pages. I have about 1000 pages for this particular site.

[Kukenan]

How long till we see the effects?

I think we are in for a major shakeup!

[Nikke]

Where is the significance in thos Mozilla-bot-thing?

Sorry for the post-and-run.
There has been speculation about the Mozilla 5 bot beeing used for picking up duplicate content and being java enabled.

I've been trying to follow it around today, and cannot see any particular pattern. I'm not positively sure, but it looks as if the Moz kind haven't visited any new pages at all. It has however been onto quite a lot of orphaned pages (not actively linked from the site any more).

I would really love to know though.

[Kukenan]

It is definitely following java...

On my site, It keeps asking for a page that is only accessible as a javascript popup window.

[incrediBILL]

I had to check - yup, googlebot/2.1 is crawling as we speak. Googles listing of my site claims it last crawled Jan 23 and they do tend to give my site a small crawl every 48 hours or so. Googlebot wasn't crawling very fast though, and my logs show they've only collected about 10,000 pages so far this month so it's only a partial crawl so far this month.

I hope they are smart enough to do a fast deep crawl outside of my prime traffic hours or this could be brutal!

[pmkpmk]

Considering the majority of members of WebmasterWorld are professionals, running typically several typically larger sites. Considering that a significant number of members has stated that GoogleBot is JUST NOW spidering their site. Assuming that an even larger number of members is following this thread and is NOT posting that the bot is crawling. Taking this all into account means that the spiders are crawling an awful lot of sites concurrently right now.

Since it is impossible to crawl the whole net at once, I keep on wondering WHAT bandwidth do those guys at Google have?

[Critter]

I believe the technical definition for Google's bandwidth is "Monster Bandwidth"

[robster124]

Errr, yeah, rather annoyingly Google is going after loads of Javascript linked pages on my site. I don't particularly want these crawled - they are plain irrelevant.

This is certainly a big advance in Google indexing technology if this proves to be the case...