homepage Welcome to WebmasterWorld Guest from 54.167.75.155
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 55 message thread spans 2 pages: 55 ( [1] 2 > >     
Deep Crawl
Googlebot on the prowl!
hoteimports




msg:731250
 12:26 am on Jan 23, 2005 (gmt 0)

Googlebot is on the prowl! Just had a very, very dep crawl. Got all 18,000 pages!

 

layer8




msg:731251
 3:27 am on Jan 24, 2005 (gmt 0)

I hope its a fashion thing, its been crawling every day but its too shy to come in so I'm waiting for a nice deep crawl.

I'll let you know if it happens

larryhatch




msg:731252
 3:44 am on Jan 24, 2005 (gmt 0)

Very same thing here on a smaller scale.
I only have 150 unique pages or so, looks like G hit them all, or maybe a few of them twice. - Larry

Livenomadic




msg:731253
 3:45 am on Jan 24, 2005 (gmt 0)

Oh man that would be sweet is I got a deep crawl, I just doubled my pages yesterday.

Spine




msg:731254
 5:50 am on Jan 24, 2005 (gmt 0)

They are doing something. Right now they are trying to crawl a bunch of long deleted pages on one of my sites.

It seems every page they are looking for is an old deleted one, and none of them are the ones acutally on my site at this moment.

Weird.

GerBot




msg:731255
 9:58 am on Jan 24, 2005 (gmt 0)

I got hit on the 23rd (all pages) which would be great if I had not uploaded my new pages on the 24th :(

bloke in a box




msg:731256
 11:21 am on Jan 24, 2005 (gmt 0)

I got hit yesterday for 3k pages which is virtually the entire site.

shabir




msg:731257
 11:21 am on Jan 24, 2005 (gmt 0)

PR went grey for all sites for me in India..

inbound




msg:731258
 2:40 pm on Jan 24, 2005 (gmt 0)

First deep crawl for one of my new sites today (UK), it took them long enough but I'm pleased.

Googlebot back from vacation and feeling ravenous.

Kukenan




msg:731259
 2:49 pm on Jan 24, 2005 (gmt 0)

Oh my God this is beautiful!

For the first time EVER I see googlebot following 301 redirects! it hits the old page and immediately after it gets the new page "200"

I have never seen it do that before!

I am going to cry! :-)

BillyS




msg:731260
 3:01 pm on Jan 24, 2005 (gmt 0)

My site was deep crawled starting last night through the early morning. Took in the entire site - around 600 pages currently.

If this follows the same pattern as the last couple of months, this means we might be several days off (Wednesday perhaps) away from a change in SERPs.

Clark




msg:731261
 3:05 pm on Jan 24, 2005 (gmt 0)

They're trying to catch all those new nofollow rel attributes. But they will have to do fairly frequent crawling because it will take time until it's implemented. So enjoy the next few months.

webdude




msg:731262
 6:18 pm on Jan 24, 2005 (gmt 0)

Yup. I am seeing the same thing, albiet very slow. One of my sites is in the process of being crawled right now. It started on Saturday and is still going on. It appears that googlebot is hitting every one of my pages. On average it looks to be one page every 7 minutes. It will come and slurp 2 pages in about 14 minutes or 1 every 7. Not like the old days when it would hit the whole site in less then an hour.

Critter




msg:731263
 11:25 pm on Jan 24, 2005 (gmt 0)

I'm seeing the exact opposite. Have 1/20th the traffic from Googlebot as usual. Very quiet (I'm scared).

:)

Critter




msg:731264
 1:15 am on Jan 25, 2005 (gmt 0)

Oops, spoke too soon.

It's baaaack.

speedshopping




msg:731265
 10:43 am on Jan 25, 2005 (gmt 0)

I am currently being crawled but googlebot is spending on average about 10 minutes on each page before moving to another one? Is this normal - my site is currently at a URL only status?

johnnie




msg:731266
 11:05 am on Jan 25, 2005 (gmt 0)

Weird, I'm being crawled in short bursts. GB hops in, crawls some 5 pages and leaves. One hour later it comes back and crawls approx. 5 others.

Are they trying to catch that new tag?

Nikke




msg:731267
 11:26 am on Jan 25, 2005 (gmt 0)

145 visits by Google's bots yesterday, and 139 of those by the one identifying itself as
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

This far, today, 44 out of 56 googlebot visits are by the Mozilla/5.0 type.

For the record: I have implemented the nofollow link attribute on most of my affiliate links.

Edited some spelling

robster124




msg:731268
 11:29 am on Jan 25, 2005 (gmt 0)

My new site is getting a deep crawl as we speak...

Nikke




msg:731269
 11:38 am on Jan 25, 2005 (gmt 0)

I went back and checked the logs for Sunday, and found that that was when the deep crawl of my site occured.
445 visits, and just 39 of them by the Mozilla/5.0 type of bot.

pmkpmk




msg:731270
 11:44 am on Jan 25, 2005 (gmt 0)

Where is the significance in thos Mozilla-bot-thing?

johnnie




msg:731271
 12:11 pm on Jan 25, 2005 (gmt 0)

I don't know... Maybe the moz-5.0 version accepts newer standards?

webdude




msg:731272
 2:29 pm on Jan 25, 2005 (gmt 0)

Mozilla/5.0 on all of mine. Still in the process of a deep crawl. Started Saturday and still not done. 25 more pages were added in the SERPs but they just show the the URL currently. Yet to pick up the titles of the pages. I have about 1000 pages for this particular site.

Kukenan




msg:731273
 3:37 pm on Jan 25, 2005 (gmt 0)

How long till we see the effects?

I think we are in for a major shakeup!

Nikke




msg:731274
 10:11 pm on Jan 25, 2005 (gmt 0)

Where is the significance in thos Mozilla-bot-thing?

Sorry for the post-and-run.
There has been speculation about the Mozilla 5 bot beeing used for picking up duplicate content and being java enabled.

I've been trying to follow it around today, and cannot see any particular pattern. I'm not positively sure, but it looks as if the Moz kind haven't visited any new pages at all. It has however been onto quite a lot of orphaned pages (not actively linked from the site any more).

I would really love to know though.

Kukenan




msg:731275
 10:59 pm on Jan 25, 2005 (gmt 0)

It is definitely following java...

On my site, It keeps asking for a page that is only accessible as a javascript popup window.

incrediBILL




msg:731276
 11:07 pm on Jan 25, 2005 (gmt 0)

I had to check - yup, googlebot/2.1 is crawling as we speak. Googles listing of my site claims it last crawled Jan 23 and they do tend to give my site a small crawl every 48 hours or so. Googlebot wasn't crawling very fast though, and my logs show they've only collected about 10,000 pages so far this month so it's only a partial crawl so far this month.

I hope they are smart enough to do a fast deep crawl outside of my prime traffic hours or this could be brutal!

pmkpmk




msg:731277
 11:11 pm on Jan 25, 2005 (gmt 0)

Considering the majority of members of WebmasterWorld are professionals, running typically several typically larger sites. Considering that a significant number of members has stated that GoogleBot is JUST NOW spidering their site. Assuming that an even larger number of members is following this thread and is NOT posting that the bot is crawling. Taking this all into account means that the spiders are crawling an awful lot of sites concurrently right now.

Since it is impossible to crawl the whole net at once, I keep on wondering WHAT bandwidth do those guys at Google have?

Critter




msg:731278
 11:15 pm on Jan 25, 2005 (gmt 0)

I believe the technical definition for Google's bandwidth is "Monster Bandwidth"

robster124




msg:731279
 11:16 pm on Jan 25, 2005 (gmt 0)

Errr, yeah, rather annoyingly Google is going after loads of Javascript linked pages on my site. I don't particularly want these crawled - they are plain irrelevant.

This is certainly a big advance in Google indexing technology if this proves to be the case...

This 55 message thread spans 2 pages: 55 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved