Welcome to WebmasterWorld Guest from 54.146.221.231

Message Too Old, No Replies

Deep Crawl

Googlebot on the prowl!

     
12:26 am on Jan 23, 2005 (gmt 0)

New User

10+ Year Member

joined:Nov 11, 2003
posts:7
votes: 0


Googlebot is on the prowl! Just had a very, very dep crawl. Got all 18,000 pages!
3:27 am on Jan 24, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Aug 7, 2003
posts:201
votes: 0


I hope its a fashion thing, its been crawling every day but its too shy to come in so I'm waiting for a nice deep crawl.

I'll let you know if it happens

3:44 am on Jan 24, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


Very same thing here on a smaller scale.
I only have 150 unique pages or so, looks like G hit them all, or maybe a few of them twice. - Larry
3:45 am on Jan 24, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Nov 3, 2003
posts:325
votes: 0


Oh man that would be sweet is I got a deep crawl, I just doubled my pages yesterday.
5:50 am on Jan 24, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Mar 6, 2003
posts:224
votes: 0


They are doing something. Right now they are trying to crawl a bunch of long deleted pages on one of my sites.

It seems every page they are looking for is an old deleted one, and none of them are the ones acutally on my site at this moment.

Weird.

9:58 am on Jan 24, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 27, 2004
posts:368
votes: 0


I got hit on the 23rd (all pages) which would be great if I had not uploaded my new pages on the 24th :(
11:21 am on Jan 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 17, 2004
posts:141
votes: 0


I got hit yesterday for 3k pages which is virtually the entire site.
11:21 am on Jan 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 23, 2001
posts:78
votes: 0


PR went grey for all sites for me in India..
2:40 pm on Jan 24, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 9, 2004
posts:1435
votes: 0


First deep crawl for one of my new sites today (UK), it took them long enough but I'm pleased.

Googlebot back from vacation and feeling ravenous.

2:49 pm on Jan 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:July 1, 2003
posts:68
votes: 0


Oh my God this is beautiful!

For the first time EVER I see googlebot following 301 redirects! it hits the old page and immediately after it gets the new page "200"

I have never seen it do that before!

I am going to cry! :-)

3:01 pm on Jan 24, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 1, 2004
posts:3181
votes: 0


My site was deep crawled starting last night through the early morning. Took in the entire site - around 600 pages currently.

If this follows the same pattern as the last couple of months, this means we might be several days off (Wednesday perhaps) away from a change in SERPs.

3:05 pm on Jan 24, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 8, 2002
posts:2335
votes: 0


They're trying to catch all those new nofollow rel attributes. But they will have to do fairly frequent crawling because it will take time until it's implemented. So enjoy the next few months.
6:18 pm on Jan 24, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 3, 2002
posts:894
votes: 0


Yup. I am seeing the same thing, albiet very slow. One of my sites is in the process of being crawled right now. It started on Saturday and is still going on. It appears that googlebot is hitting every one of my pages. On average it looks to be one page every 7 minutes. It will come and slurp 2 pages in about 14 minutes or 1 every 7. Not like the old days when it would hit the whole site in less then an hour.
11:25 pm on Jan 24, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 31, 2003
posts:386
votes: 0


I'm seeing the exact opposite. Have 1/20th the traffic from Googlebot as usual. Very quiet (I'm scared).

:)

1:15 am on Jan 25, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 31, 2003
posts:386
votes: 0


Oops, spoke too soon.

It's baaaack.

10:43 am on Jan 25, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 10, 2005
posts:124
votes: 0


I am currently being crawled but googlebot is spending on average about 10 minutes on each page before moving to another one? Is this normal - my site is currently at a URL only status?
11:05 am on Jan 25, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 17, 2004
posts:1354
votes: 0


Weird, I'm being crawled in short bursts. GB hops in, crawls some 5 pages and leaves. One hour later it comes back and crawls approx. 5 others.

Are they trying to catch that new tag?

11:26 am on Jan 25, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:May 13, 2003
posts:442
votes: 0


145 visits by Google's bots yesterday, and 139 of those by the one identifying itself as
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

This far, today, 44 out of 56 googlebot visits are by the Mozilla/5.0 type.

For the record: I have implemented the nofollow link attribute on most of my affiliate links.

Edited some spelling

11:29 am on Jan 25, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 2, 2004
posts:125
votes: 0


My new site is getting a deep crawl as we speak...
11:38 am on Jan 25, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:May 13, 2003
posts:442
votes: 0


I went back and checked the logs for Sunday, and found that that was when the deep crawl of my site occured.
445 visits, and just 39 of them by the Mozilla/5.0 type of bot.
11:44 am on Jan 25, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 11, 2002
posts:2024
votes: 0


Where is the significance in thos Mozilla-bot-thing?
12:11 pm on Jan 25, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 17, 2004
posts:1354
votes: 0


I don't know... Maybe the moz-5.0 version accepts newer standards?
2:29 pm on Jan 25, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 3, 2002
posts:894
votes: 0


Mozilla/5.0 on all of mine. Still in the process of a deep crawl. Started Saturday and still not done. 25 more pages were added in the SERPs but they just show the the URL currently. Yet to pick up the titles of the pages. I have about 1000 pages for this particular site.
3:37 pm on Jan 25, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:July 1, 2003
posts:68
votes: 0


How long till we see the effects?

I think we are in for a major shakeup!

10:11 pm on Jan 25, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:May 13, 2003
posts:442
votes: 0


Where is the significance in thos Mozilla-bot-thing?

Sorry for the post-and-run.
There has been speculation about the Mozilla 5 bot beeing used for picking up duplicate content and being java enabled.

I've been trying to follow it around today, and cannot see any particular pattern. I'm not positively sure, but it looks as if the Moz kind haven't visited any new pages at all. It has however been onto quite a lot of orphaned pages (not actively linked from the site any more).

I would really love to know though.

10:59 pm on Jan 25, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:July 1, 2003
posts:68
votes: 0


It is definitely following java...

On my site, It keeps asking for a page that is only accessible as a javascript popup window.

11:07 pm on Jan 25, 2005 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14622
votes: 87


I had to check - yup, googlebot/2.1 is crawling as we speak. Googles listing of my site claims it last crawled Jan 23 and they do tend to give my site a small crawl every 48 hours or so. Googlebot wasn't crawling very fast though, and my logs show they've only collected about 10,000 pages so far this month so it's only a partial crawl so far this month.

I hope they are smart enough to do a fast deep crawl outside of my prime traffic hours or this could be brutal!

11:11 pm on Jan 25, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 11, 2002
posts:2024
votes: 0


Considering the majority of members of WebmasterWorld are professionals, running typically several typically larger sites. Considering that a significant number of members has stated that GoogleBot is JUST NOW spidering their site. Assuming that an even larger number of members is following this thread and is NOT posting that the bot is crawling. Taking this all into account means that the spiders are crawling an awful lot of sites concurrently right now.

Since it is impossible to crawl the whole net at once, I keep on wondering WHAT bandwidth do those guys at Google have?

11:15 pm on Jan 25, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 31, 2003
posts:386
votes: 0


I believe the technical definition for Google's bandwidth is "Monster Bandwidth"
11:16 pm on Jan 25, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 2, 2004
posts:125
votes: 0


Errr, yeah, rather annoyingly Google is going after loads of Javascript linked pages on my site. I don't particularly want these crawled - they are plain irrelevant.

This is certainly a big advance in Google indexing technology if this proves to be the case...

This 55 message thread spans 2 pages: 55