Welcome to WebmasterWorld Guest from 54.196.214.35

Message Too Old, No Replies

What can I learn from Googlebot activity in site logs?

     
5:00 pm on Oct 14, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Mar 15, 2004
posts:241
votes: 0


Hi,

I've been looking at some of my site logs recently and wondering whether there's anything I can glean from them other than "oh, Googlebot's picked up my new pages" or "Wow, Googlebot's requesting a lot of pages today". Is it possible to infer anything from frequency or order that G accesses the pages? Repeated requests for a single page? Does the request order tell me anything about my link structure? I presume G works on some sort of logical basis, hence my query.

Apologies in advance if this is a noob subject...!

All comments welcome.

5:59 am on Oct 15, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


Shameless bump here -- I'd also be very interested in any conclusions people have drawn from Googlebot patterns.
8:35 am on Oct 17, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Mar 15, 2004
posts:241
votes: 0


Come on folks, has nobody looked at this before?
9:11 am on Oct 18, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 25, 2003
posts:365
votes: 0


Well, I don't know if that's of some value for you:

Gbot comes daily and requests robots.txt and root - at some busy sites he comes twice a day.

At least once every week or so he crawls deep through my sites visiting a lot of pages.

Constantly during those visits he tries to get some that doesn't exist:
GET /sortdir/asc HTTP/1.0
GET /eText/ HTTP/1.0
GET /eCat2/ HTTP/1.0
... always similar although 404ed.

New pages are best fed to him by deep links from home (better than from site map).

It takes max. 3 days for new pages to appear in the index after being crawled.