Googlebot's multi-hits of same page is such a waste

seeing tons of duplicate pages hit by different gbot ips

[Clark]

Ever since Dominic and the freshdeepbot, I've been seeing Googlebot's different IPS hitting the same pages over and over and over again. Sometimes 3-4 or 5 IPs hitting the same page within seconds of each other. And a daily refresh of the very same pages which haven't changed for years.

Google's crawlers still need a lot of work to coordinate their effort better and work in harmony instead of this way.

It's like having a team of engineers designing a revolutionary car together, but instead of giving each members a different part of the engine to work on, they are all designing the same parts simultaneously and then beginning the same project again the next day.

It reminds me of "GroundHog Day".

[WebGuerrilla]

I get a kick out of watching pages that haven't been updated in over 2 years continue to get fresh dates added every other day.

It certainly does seem like they have a long way to go before the fresh system actually produces the intended results.

[nancyb]

and I have the opposite - googlebot visits the same pages sometimes 4-5 times a month, sees updated pages but never gives a fresh date anymore - almost 7 months since the last one.

Thankfully, s/he does add the new pages though, and quickly w/o a freshdate - velly velly strange behaviour ;)

[GoogleGuy]

WebGuerrilla, I think the fresh dates just mean that the page was recently crawled--it doesn't mean that the page has changed recently.

Clark, a crawl engineer asked me if you could do a spam report and mention a few urls that are getting hit multiple times--they want to see if anything needs debugging. Would you mind doing a quick report and mentioning your login name? I'll pass on the urls for someone to check on what's going on with the multiple fetches if you're willing.

[nancyb]

now, I'm really confused.

I think the fresh dates just mean that the page was recently crawled--it doesn't mean that the page has changed recently.

so, does this mean that because my pages are updated several times a month and crawled several times a month that is why I haven't seen a fresh date in 7 months?

Not that it's terribly important, just curious.