Is Googlebot Deep Crawling?

I've not seen much activity yet

[Kerrin]

Ayone seeing any deep crawling from Googlebot? I want to make some major changes to my website but, just to be safe, I don't want to do it if the deep crawl has already started.

[warmasol]

No, I saw the "fresh"-crawler every day, but no "deep"-crawler. I think he is a little too late.

[Grumpus]

Nope. Just the FreshBot here, too.

G.

[MrLucky]

same here

[Kerrin]

Thanks guys, i'd better get busy ;)

[lipskin]

Googlebot finally started deep crawling one of my PR5 sites about an hour ago, but it's the only site to get deep crawled so far.

[warmasol]

Now I see the "deep"-crawler(216.239.46.*) on my sites. I think he starts his monthly crawl.

[Slade]

Is there any consistancy to these references of "FreshBot" and "DeepCrawler"?

Specifically, IP ranges?

[warmasol]

FreshBot:
64.68.82.*

DeepCrawler:
216.239.46.*

[Rugles]

Deep crawling here for the last 12 hours.

[savvy1]

Seeing activity from both, seems more from the fresh bots

[MrLucky]

Activity has picked up over here -- looking like a deep crawl.

Any difference in useragent between the freshbot and deepcrawler? Or only by IP?

[jdMorgan]

Deepcrawling started 16:40:54 (-0400) on one of mine - index.html is PR5.

I think the fresh crawler reverse resolves to "crawlernn.googlebot.com" and the deepcrawler to "crawlnn.googlebot.com", but both UAs are the same.

Jim

[KakenBetaal]

Started on my PR5 site yesterday morning. jdMorgan, I'd tend to agree on the crawl vs crawler theory. Back in Marchish, crawler used to be the imagebot crawler, IIRC.

[dukeblue219]

She just arrived on scene at 5 AM this morning, and proceeded to crawl about 200 pages in half an hour. Sounds like a good deep crawl to me.

[Mr_Tickle]

Deep crawling my site right now.

IP address is 64.68.82.*

Made my site changes just in time :)

[Grumpus]

MrTickle - that's the Freshbot you're seeing. Check google in about 24 hours (or less) and you'll see those pages in the index, most likely.

G.

[Mr_Tickle]

Cool, that means Freshbot is deepcrawling my site now :)

[KakenBetaal]

Could be, Grumpus, but to me it looks like both ranges are being used for deep crawling:

From 64.68.x.y: 2273 pages requested

From 216.239.x.y: 4608 pages requested

They are fairly evenly mixed up throughout my logfile. I'm not so sure on what I said in a previous message in this thread now.

I only have a meagre little PR5 site with about 250 flat html pages and a phpBB forum, so getting so many pages requested by the fresh crawl seems unlikely.

[promis]

64.68.82.* has been deep crawling my site yesterday October 4th.
Have not seen the pages crawled yet on any of www, www2 or www3.
Any idea how long it takes for them to appear?

[Slade]

I have PR 1 or 2 personal site (pretty much non-existant backlinks) that got deepcrawled starting at 5:30 on Oct 4. Still getting hits actually, even though there are only about 2 dozen pages.

[martin]

Started on 2nd of October, a little re-fresh on 4th, and back to crawl on the 5th.

[vitaplease]

Just checked my log-stats,

Is it me, or was this deep-crawl one of the most intense in months?

[Visit Thailand]

Just curious but do you think it could damage anything if your are constantly uploading pages old and new to the site when googlebots sniffing around your site?

[Grumpus]

(From higher up the thread) Cool that the freshbot looks to be going deep on some sites. That could mean a much deeper crawl this month. :)

Uploading pages during the crawl won't hurt anything. If you change a page after it's been crawled, though, Googlebot likely won't find any new links on it and the cached version after the next update won't reflect the newer content. Other than that, I wouldn't worry. If you've got something new to say, say it!

G.

[thunderpaste]

I know this is a bit off topic, but what log analysis software do you guys prefer for seeing what pages googlebot is crawling?

[Sasquatch]

grep to see everything google is doing.

For my real content pages (the ones that I want crawled) I have PHP code that logs the info I want to a mysql db when I get hit with a UA of a SE spider. Then I have a page to do some manipulation of this info on my admin page. It works really slick.

[crobb305]

Are you guys referring to the deep crawl for the October update? I am not sure how far in advance the deep crawls occur.

Also, what ip(s) should I be looking for in my logs?

Thanks :)

[gilli]

Try looking here:
[searchengineworld.com...] . Don't know how uptodate this is. Also try search these forums for "google ip addresses" or similar.