homepage Welcome to WebmasterWorld Guest from 54.226.80.55
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Bot on the Hard Prowl
Gbot running hard, fast, and deep.
Brett_Tabke




msg:708975
 5:44 am on Feb 5, 2005 (gmt 0)

Just hammering some sites right now.

1pv a second here - another site was 10 pvs a second.

 

danny




msg:708976
 6:19 am on Feb 5, 2005 (gmt 0)

Googlebot has fetched pretty much every one of my web pages every day, for the last five years or more.

I don't know why it needs to spider them that often, since they're almost entirely static. A weekly fetch for most pages would work fine; with daily spidering for the few pages where links to new pages appear.

Fortunately my small-medium static sites cope just fine. I shudder to think what search-engine spidering does to a big database-backed site like this, though.

pendanticist




msg:708977
 6:22 am on Feb 5, 2005 (gmt 0)

Well, today is the first time I've ever seen Google handed 304s all the way down the line.

larryhatch




msg:708978
 6:28 am on Feb 5, 2005 (gmt 0)

Dumb question:

What does a returned "304" mean exactly in this spidering context?
Is it a message indicating "You already spidered this page, it hasn't changed any yet"?
.. or am I missing something?
I'm starting to see 304s here too. - Larry

Powdork




msg:708979
 6:44 am on Feb 5, 2005 (gmt 0)

Is it a message indicating "You already spidered this page, it hasn't changed any yet"?
Thats pretty much it. It is the Not Modified Since header.

Danny, Is your server set up to return the 304 header when appropriate?

danny




msg:708980
 6:47 am on Feb 5, 2005 (gmt 0)

Danny, Is your server set up to return the 304 header when appropriate?

Yep. My logs show lots of 304s.

Doesn't stop Googlebot coming back for those pages again the next day.

Powdork




msg:708981
 7:16 am on Feb 5, 2005 (gmt 0)

No it will still come back but when it gets the 304 thats all its supposed to get. When it gets a 200 there will also be a number indicating the number of bytes of the downloaded file. This number should not exist with a 304.

walkman




msg:708982
 7:25 am on Feb 5, 2005 (gmt 0)

same here. Gbot gone wild. I noticed that they do this all the time after a PR and /or backlink update.

Galtego




msg:708983
 7:41 am on Feb 5, 2005 (gmt 0)

I'm seeing a lot more of Googlebot-Image than usual.

larryhatch




msg:708984
 7:41 am on Feb 5, 2005 (gmt 0)

Is it a message indicating "You already spidered this page"?

Thats pretty much it. It is the Not Modified Since header.

Thanks Powdork, that's what I thought. I'm going to look at access_logs again.
Maybe I'll learn something new about G spidering. -Larry

Powdork




msg:708985
 7:46 am on Feb 5, 2005 (gmt 0)

I noticed the post which linked to w3c.org is now gone. Is that sort of link no longer ok?
I'm not complaining, just asking.

pendanticist




msg:708986
 8:21 am on Feb 5, 2005 (gmt 0)

I deleted my post because of you. The mods deleted it because it was empty.

Powdork




msg:708987
 8:27 am on Feb 5, 2005 (gmt 0)

Sorry about that. I took a break while posting and you posted yours in the meantime. Yours was better.

danny




msg:708988
 9:28 am on Feb 5, 2005 (gmt 0)

I guess it makes sense for Google to keep checking 304 pages daily, though some kind of back-off would seem obvious.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved