Welcome to WebmasterWorld Guest from 54.227.214.52

Forum Moderators: open

Is Googlebot Deep Crawling?

I've not seen much activity yet

   
5:25 pm on Oct 1, 2002 (gmt 0)

10+ Year Member



Ayone seeing any deep crawling from Googlebot? I want to make some major changes to my website but, just to be safe, I don't want to do it if the deep crawl has already started.
5:44 pm on Oct 1, 2002 (gmt 0)

10+ Year Member



No, I saw the "fresh"-crawler every day, but no "deep"-crawler. I think he is a little too late.
6:08 pm on Oct 1, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Nope. Just the FreshBot here, too.

G.

6:13 pm on Oct 1, 2002 (gmt 0)

10+ Year Member



same here
6:25 pm on Oct 1, 2002 (gmt 0)

10+ Year Member



Thanks guys, i'd better get busy ;)
6:42 pm on Oct 1, 2002 (gmt 0)

10+ Year Member



Googlebot finally started deep crawling one of my PR5 sites about an hour ago, but it's the only site to get deep crawled so far.
1:37 pm on Oct 2, 2002 (gmt 0)

10+ Year Member



Now I see the "deep"-crawler(216.239.46.*) on my sites. I think he starts his monthly crawl.
2:22 pm on Oct 2, 2002 (gmt 0)

10+ Year Member



Is there any consistancy to these references of "FreshBot" and "DeepCrawler"?

Specifically, IP ranges?

2:29 pm on Oct 2, 2002 (gmt 0)

10+ Year Member



FreshBot:
64.68.82.*

DeepCrawler:
216.239.46.*

2:33 pm on Oct 2, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Deep crawling here for the last 12 hours.
2:33 pm on Oct 2, 2002 (gmt 0)

10+ Year Member



Seeing activity from both, seems more from the fresh bots
8:52 pm on Oct 2, 2002 (gmt 0)

10+ Year Member



Activity has picked up over here -- looking like a deep crawl.

Any difference in useragent between the freshbot and deepcrawler? Or only by IP?

9:46 pm on Oct 2, 2002 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Deepcrawling started 16:40:54 (-0400) on one of mine - index.html is PR5.

I think the fresh crawler reverse resolves to "crawlernn.googlebot.com" and the deepcrawler to "crawlnn.googlebot.com", but both UAs are the same.

Jim

9:16 am on Oct 3, 2002 (gmt 0)

10+ Year Member



Started on my PR5 site yesterday morning. jdMorgan, I'd tend to agree on the crawl vs crawler theory. Back in Marchish, crawler used to be the imagebot crawler, IIRC.
10:58 am on Oct 3, 2002 (gmt 0)

10+ Year Member



She just arrived on scene at 5 AM this morning, and proceeded to crawl about 200 pages in half an hour. Sounds like a good deep crawl to me.
12:23 pm on Oct 3, 2002 (gmt 0)

10+ Year Member



Deep crawling my site right now.

IP address is 64.68.82.*

Made my site changes just in time :)

12:27 pm on Oct 3, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



MrTickle - that's the Freshbot you're seeing. Check google in about 24 hours (or less) and you'll see those pages in the index, most likely.

G.

5:10 pm on Oct 3, 2002 (gmt 0)

10+ Year Member



Cool, that means Freshbot is deepcrawling my site now :)
10:23 am on Oct 4, 2002 (gmt 0)

10+ Year Member



Could be, Grumpus, but to me it looks like both ranges are being used for deep crawling:

From 64.68.x.y: 2273 pages requested

From 216.239.x.y: 4608 pages requested

They are fairly evenly mixed up throughout my logfile. I'm not so sure on what I said in a previous message in this thread now.

I only have a meagre little PR5 site with about 250 flat html pages and a phpBB forum, so getting so many pages requested by the fresh crawl seems unlikely.

9:18 am on Oct 5, 2002 (gmt 0)

10+ Year Member



64.68.82.* has been deep crawling my site yesterday October 4th.
Have not seen the pages crawled yet on any of www, www2 or www3.
Any idea how long it takes for them to appear?
7:02 pm on Oct 5, 2002 (gmt 0)

10+ Year Member



I have PR 1 or 2 personal site (pretty much non-existant backlinks) that got deepcrawled starting at 5:30 on Oct 4. Still getting hits actually, even though there are only about 2 dozen pages.
9:25 pm on Oct 5, 2002 (gmt 0)

10+ Year Member



Started on 2nd of October, a little re-fresh on 4th, and back to crawl on the 5th.
8:52 am on Oct 7, 2002 (gmt 0)

WebmasterWorld Senior Member vitaplease is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Just checked my log-stats,

Is it me, or was this deep-crawl one of the most intense in months?

9:51 am on Oct 7, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Just curious but do you think it could damage anything if your are constantly uploading pages old and new to the site when googlebots sniffing around your site?
11:40 am on Oct 7, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



(From higher up the thread) Cool that the freshbot looks to be going deep on some sites. That could mean a much deeper crawl this month. :)

Uploading pages during the crawl won't hurt anything. If you change a page after it's been crawled, though, Googlebot likely won't find any new links on it and the cached version after the next update won't reflect the newer content. Other than that, I wouldn't worry. If you've got something new to say, say it!

G.

5:15 pm on Oct 8, 2002 (gmt 0)

10+ Year Member



I know this is a bit off topic, but what log analysis software do you guys prefer for seeing what pages googlebot is crawling?
5:48 pm on Oct 8, 2002 (gmt 0)

10+ Year Member



grep to see everything google is doing.

For my real content pages (the ones that I want crawled) I have PHP code that logs the info I want to a mysql db when I get hit with a UA of a SE spider. Then I have a page to do some manipulation of this info on my admin page. It works really slick.

4:53 am on Oct 9, 2002 (gmt 0)

WebmasterWorld Senior Member crobb305 is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Are you guys referring to the deep crawl for the October update? I am not sure how far in advance the deep crawls occur.

Also, what ip(s) should I be looking for in my logs?

Thanks :)

8:15 am on Oct 9, 2002 (gmt 0)

10+ Year Member



Try looking here:
[searchengineworld.com...] . Don't know how uptodate this is. Also try search these forums for "google ip addresses" or similar.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month