homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 55 message thread spans 2 pages: 55 ( [1] 2 > >     
Deep Crawl
Googlebot on the prowl!

 12:26 am on Jan 23, 2005 (gmt 0)

Googlebot is on the prowl! Just had a very, very dep crawl. Got all 18,000 pages!



 3:27 am on Jan 24, 2005 (gmt 0)

I hope its a fashion thing, its been crawling every day but its too shy to come in so I'm waiting for a nice deep crawl.

I'll let you know if it happens


 3:44 am on Jan 24, 2005 (gmt 0)

Very same thing here on a smaller scale.
I only have 150 unique pages or so, looks like G hit them all, or maybe a few of them twice. - Larry


 3:45 am on Jan 24, 2005 (gmt 0)

Oh man that would be sweet is I got a deep crawl, I just doubled my pages yesterday.


 5:50 am on Jan 24, 2005 (gmt 0)

They are doing something. Right now they are trying to crawl a bunch of long deleted pages on one of my sites.

It seems every page they are looking for is an old deleted one, and none of them are the ones acutally on my site at this moment.



 9:58 am on Jan 24, 2005 (gmt 0)

I got hit on the 23rd (all pages) which would be great if I had not uploaded my new pages on the 24th :(

bloke in a box

 11:21 am on Jan 24, 2005 (gmt 0)

I got hit yesterday for 3k pages which is virtually the entire site.


 11:21 am on Jan 24, 2005 (gmt 0)

PR went grey for all sites for me in India..


 2:40 pm on Jan 24, 2005 (gmt 0)

First deep crawl for one of my new sites today (UK), it took them long enough but I'm pleased.

Googlebot back from vacation and feeling ravenous.


 2:49 pm on Jan 24, 2005 (gmt 0)

Oh my God this is beautiful!

For the first time EVER I see googlebot following 301 redirects! it hits the old page and immediately after it gets the new page "200"

I have never seen it do that before!

I am going to cry! :-)


 3:01 pm on Jan 24, 2005 (gmt 0)

My site was deep crawled starting last night through the early morning. Took in the entire site - around 600 pages currently.

If this follows the same pattern as the last couple of months, this means we might be several days off (Wednesday perhaps) away from a change in SERPs.


 3:05 pm on Jan 24, 2005 (gmt 0)

They're trying to catch all those new nofollow rel attributes. But they will have to do fairly frequent crawling because it will take time until it's implemented. So enjoy the next few months.


 6:18 pm on Jan 24, 2005 (gmt 0)

Yup. I am seeing the same thing, albiet very slow. One of my sites is in the process of being crawled right now. It started on Saturday and is still going on. It appears that googlebot is hitting every one of my pages. On average it looks to be one page every 7 minutes. It will come and slurp 2 pages in about 14 minutes or 1 every 7. Not like the old days when it would hit the whole site in less then an hour.


 11:25 pm on Jan 24, 2005 (gmt 0)

I'm seeing the exact opposite. Have 1/20th the traffic from Googlebot as usual. Very quiet (I'm scared).



 1:15 am on Jan 25, 2005 (gmt 0)

Oops, spoke too soon.

It's baaaack.


 10:43 am on Jan 25, 2005 (gmt 0)

I am currently being crawled but googlebot is spending on average about 10 minutes on each page before moving to another one? Is this normal - my site is currently at a URL only status?


 11:05 am on Jan 25, 2005 (gmt 0)

Weird, I'm being crawled in short bursts. GB hops in, crawls some 5 pages and leaves. One hour later it comes back and crawls approx. 5 others.

Are they trying to catch that new tag?


 11:26 am on Jan 25, 2005 (gmt 0)

145 visits by Google's bots yesterday, and 139 of those by the one identifying itself as
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

This far, today, 44 out of 56 googlebot visits are by the Mozilla/5.0 type.

For the record: I have implemented the nofollow link attribute on most of my affiliate links.

Edited some spelling


 11:29 am on Jan 25, 2005 (gmt 0)

My new site is getting a deep crawl as we speak...


 11:38 am on Jan 25, 2005 (gmt 0)

I went back and checked the logs for Sunday, and found that that was when the deep crawl of my site occured.
445 visits, and just 39 of them by the Mozilla/5.0 type of bot.


 11:44 am on Jan 25, 2005 (gmt 0)

Where is the significance in thos Mozilla-bot-thing?


 12:11 pm on Jan 25, 2005 (gmt 0)

I don't know... Maybe the moz-5.0 version accepts newer standards?


 2:29 pm on Jan 25, 2005 (gmt 0)

Mozilla/5.0 on all of mine. Still in the process of a deep crawl. Started Saturday and still not done. 25 more pages were added in the SERPs but they just show the the URL currently. Yet to pick up the titles of the pages. I have about 1000 pages for this particular site.


 3:37 pm on Jan 25, 2005 (gmt 0)

How long till we see the effects?

I think we are in for a major shakeup!


 10:11 pm on Jan 25, 2005 (gmt 0)

Where is the significance in thos Mozilla-bot-thing?

Sorry for the post-and-run.
There has been speculation about the Mozilla 5 bot beeing used for picking up duplicate content and being java enabled.

I've been trying to follow it around today, and cannot see any particular pattern. I'm not positively sure, but it looks as if the Moz kind haven't visited any new pages at all. It has however been onto quite a lot of orphaned pages (not actively linked from the site any more).

I would really love to know though.


 10:59 pm on Jan 25, 2005 (gmt 0)

It is definitely following java...

On my site, It keeps asking for a page that is only accessible as a javascript popup window.


 11:07 pm on Jan 25, 2005 (gmt 0)

I had to check - yup, googlebot/2.1 is crawling as we speak. Googles listing of my site claims it last crawled Jan 23 and they do tend to give my site a small crawl every 48 hours or so. Googlebot wasn't crawling very fast though, and my logs show they've only collected about 10,000 pages so far this month so it's only a partial crawl so far this month.

I hope they are smart enough to do a fast deep crawl outside of my prime traffic hours or this could be brutal!


 11:11 pm on Jan 25, 2005 (gmt 0)

Considering the majority of members of WebmasterWorld are professionals, running typically several typically larger sites. Considering that a significant number of members has stated that GoogleBot is JUST NOW spidering their site. Assuming that an even larger number of members is following this thread and is NOT posting that the bot is crawling. Taking this all into account means that the spiders are crawling an awful lot of sites concurrently right now.

Since it is impossible to crawl the whole net at once, I keep on wondering WHAT bandwidth do those guys at Google have?


 11:15 pm on Jan 25, 2005 (gmt 0)

I believe the technical definition for Google's bandwidth is "Monster Bandwidth"


 11:16 pm on Jan 25, 2005 (gmt 0)

Errr, yeah, rather annoyingly Google is going after loads of Javascript linked pages on my site. I don't particularly want these crawled - they are plain irrelevant.

This is certainly a big advance in Google indexing technology if this proves to be the case...

This 55 message thread spans 2 pages: 55 ( [1] 2 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved