Bot on the Hard Prowl

Gbot running hard, fast, and deep.

[Brett_Tabke]

Just hammering some sites right now.

1pv a second here - another site was 10 pvs a second.

[danny]

Googlebot has fetched pretty much every one of my web pages every day, for the last five years or more.

I don't know why it needs to spider them that often, since they're almost entirely static. A weekly fetch for most pages would work fine; with daily spidering for the few pages where links to new pages appear.

Fortunately my small-medium static sites cope just fine. I shudder to think what search-engine spidering does to a big database-backed site like this, though.

[pendanticist]

Well, today is the first time I've ever seen Google handed 304s all the way down the line.

[larryhatch]

Dumb question:

What does a returned "304" mean exactly in this spidering context?
Is it a message indicating "You already spidered this page, it hasn't changed any yet"?
.. or am I missing something?
I'm starting to see 304s here too. - Larry

[Powdork]

Is it a message indicating "You already spidered this page, it hasn't changed any yet"?

Thats pretty much it. It is the Not Modified Since header.

Danny, Is your server set up to return the 304 header when appropriate?

[danny]

Danny, Is your server set up to return the 304 header when appropriate?

Yep. My logs show lots of 304s.

Doesn't stop Googlebot coming back for those pages again the next day.

[Powdork]

No it will still come back but when it gets the 304 thats all its supposed to get. When it gets a 200 there will also be a number indicating the number of bytes of the downloaded file. This number should not exist with a 304.

[walkman]

same here. Gbot gone wild. I noticed that they do this all the time after a PR and /or backlink update.

[Galtego]

I'm seeing a lot more of Googlebot-Image than usual.

[larryhatch]

Is it a message indicating "You already spidered this page"?

Thats pretty much it. It is the Not Modified Since header.

Thanks Powdork, that's what I thought. I'm going to look at access_logs again.
Maybe I'll learn something new about G spidering. -Larry

[Powdork]

I noticed the post which linked to w3c.org is now gone. Is that sort of link no longer ok?
I'm not complaining, just asking.

[pendanticist]

I deleted my post because of you. The mods deleted it because it was empty.

[Powdork]

Sorry about that. I took a break while posting and you posted yours in the meantime. Yours was better.

[danny]

I guess it makes sense for Google to keep checking 304 pages daily, though some kind of back-off would seem obvious.