Welcome to WebmasterWorld Guest from 54.225.31.78

Message Too Old, No Replies

Deep Crawl

Googlebot on the prowl!

     
12:26 am on Jan 23, 2005 (gmt 0)

10+ Year Member



Googlebot is on the prowl! Just had a very, very dep crawl. Got all 18,000 pages!
3:27 am on Jan 24, 2005 (gmt 0)

10+ Year Member



I hope its a fashion thing, its been crawling every day but its too shy to come in so I'm waiting for a nice deep crawl.

I'll let you know if it happens

3:44 am on Jan 24, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Very same thing here on a smaller scale.
I only have 150 unique pages or so, looks like G hit them all, or maybe a few of them twice. - Larry
3:45 am on Jan 24, 2005 (gmt 0)

10+ Year Member



Oh man that would be sweet is I got a deep crawl, I just doubled my pages yesterday.
5:50 am on Jan 24, 2005 (gmt 0)

10+ Year Member



They are doing something. Right now they are trying to crawl a bunch of long deleted pages on one of my sites.

It seems every page they are looking for is an old deleted one, and none of them are the ones acutally on my site at this moment.

Weird.

9:58 am on Jan 24, 2005 (gmt 0)

10+ Year Member



I got hit on the 23rd (all pages) which would be great if I had not uploaded my new pages on the 24th :(
11:21 am on Jan 24, 2005 (gmt 0)

10+ Year Member



I got hit yesterday for 3k pages which is virtually the entire site.
11:21 am on Jan 24, 2005 (gmt 0)

10+ Year Member



PR went grey for all sites for me in India..
2:40 pm on Jan 24, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



First deep crawl for one of my new sites today (UK), it took them long enough but I'm pleased.

Googlebot back from vacation and feeling ravenous.

2:49 pm on Jan 24, 2005 (gmt 0)

10+ Year Member



Oh my God this is beautiful!

For the first time EVER I see googlebot following 301 redirects! it hits the old page and immediately after it gets the new page "200"

I have never seen it do that before!

I am going to cry! :-)

3:01 pm on Jan 24, 2005 (gmt 0)

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member



My site was deep crawled starting last night through the early morning. Took in the entire site - around 600 pages currently.

If this follows the same pattern as the last couple of months, this means we might be several days off (Wednesday perhaps) away from a change in SERPs.

3:05 pm on Jan 24, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



They're trying to catch all those new nofollow rel attributes. But they will have to do fairly frequent crawling because it will take time until it's implemented. So enjoy the next few months.
6:18 pm on Jan 24, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yup. I am seeing the same thing, albiet very slow. One of my sites is in the process of being crawled right now. It started on Saturday and is still going on. It appears that googlebot is hitting every one of my pages. On average it looks to be one page every 7 minutes. It will come and slurp 2 pages in about 14 minutes or 1 every 7. Not like the old days when it would hit the whole site in less then an hour.
11:25 pm on Jan 24, 2005 (gmt 0)

10+ Year Member



I'm seeing the exact opposite. Have 1/20th the traffic from Googlebot as usual. Very quiet (I'm scared).

:)

1:15 am on Jan 25, 2005 (gmt 0)

10+ Year Member



Oops, spoke too soon.

It's baaaack.

10:43 am on Jan 25, 2005 (gmt 0)

10+ Year Member



I am currently being crawled but googlebot is spending on average about 10 minutes on each page before moving to another one? Is this normal - my site is currently at a URL only status?
11:05 am on Jan 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Weird, I'm being crawled in short bursts. GB hops in, crawls some 5 pages and leaves. One hour later it comes back and crawls approx. 5 others.

Are they trying to catch that new tag?

11:26 am on Jan 25, 2005 (gmt 0)

10+ Year Member



145 visits by Google's bots yesterday, and 139 of those by the one identifying itself as
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

This far, today, 44 out of 56 googlebot visits are by the Mozilla/5.0 type.

For the record: I have implemented the nofollow link attribute on most of my affiliate links.

Edited some spelling

11:29 am on Jan 25, 2005 (gmt 0)

10+ Year Member



My new site is getting a deep crawl as we speak...
11:38 am on Jan 25, 2005 (gmt 0)

10+ Year Member



I went back and checked the logs for Sunday, and found that that was when the deep crawl of my site occured.
445 visits, and just 39 of them by the Mozilla/5.0 type of bot.
11:44 am on Jan 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Where is the significance in thos Mozilla-bot-thing?
12:11 pm on Jan 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I don't know... Maybe the moz-5.0 version accepts newer standards?
2:29 pm on Jan 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Mozilla/5.0 on all of mine. Still in the process of a deep crawl. Started Saturday and still not done. 25 more pages were added in the SERPs but they just show the the URL currently. Yet to pick up the titles of the pages. I have about 1000 pages for this particular site.
3:37 pm on Jan 25, 2005 (gmt 0)

10+ Year Member



How long till we see the effects?

I think we are in for a major shakeup!

10:11 pm on Jan 25, 2005 (gmt 0)

10+ Year Member



Where is the significance in thos Mozilla-bot-thing?

Sorry for the post-and-run.
There has been speculation about the Mozilla 5 bot beeing used for picking up duplicate content and being java enabled.

I've been trying to follow it around today, and cannot see any particular pattern. I'm not positively sure, but it looks as if the Moz kind haven't visited any new pages at all. It has however been onto quite a lot of orphaned pages (not actively linked from the site any more).

I would really love to know though.

10:59 pm on Jan 25, 2005 (gmt 0)

10+ Year Member



It is definitely following java...

On my site, It keeps asking for a page that is only accessible as a javascript popup window.

11:07 pm on Jan 25, 2005 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I had to check - yup, googlebot/2.1 is crawling as we speak. Googles listing of my site claims it last crawled Jan 23 and they do tend to give my site a small crawl every 48 hours or so. Googlebot wasn't crawling very fast though, and my logs show they've only collected about 10,000 pages so far this month so it's only a partial crawl so far this month.

I hope they are smart enough to do a fast deep crawl outside of my prime traffic hours or this could be brutal!

11:11 pm on Jan 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Considering the majority of members of WebmasterWorld are professionals, running typically several typically larger sites. Considering that a significant number of members has stated that GoogleBot is JUST NOW spidering their site. Assuming that an even larger number of members is following this thread and is NOT posting that the bot is crawling. Taking this all into account means that the spiders are crawling an awful lot of sites concurrently right now.

Since it is impossible to crawl the whole net at once, I keep on wondering WHAT bandwidth do those guys at Google have?

11:15 pm on Jan 25, 2005 (gmt 0)

10+ Year Member



I believe the technical definition for Google's bandwidth is "Monster Bandwidth"
11:16 pm on Jan 25, 2005 (gmt 0)

10+ Year Member



Errr, yeah, rather annoyingly Google is going after loads of Javascript linked pages on my site. I don't particularly want these crawled - they are plain irrelevant.

This is certainly a big advance in Google indexing technology if this proves to be the case...

This 55 message thread spans 2 pages: 55
 

Featured Threads

Hot Threads This Week

Hot Threads This Month