Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

What can I learn from Googlebot activity in site logs?



5:00 pm on Oct 14, 2005 (gmt 0)

10+ Year Member


I've been looking at some of my site logs recently and wondering whether there's anything I can glean from them other than "oh, Googlebot's picked up my new pages" or "Wow, Googlebot's requesting a lot of pages today". Is it possible to infer anything from frequency or order that G accesses the pages? Repeated requests for a single page? Does the request order tell me anything about my link structure? I presume G works on some sort of logical basis, hence my query.

Apologies in advance if this is a noob subject...!

All comments welcome.


5:59 am on Oct 15, 2005 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Shameless bump here -- I'd also be very interested in any conclusions people have drawn from Googlebot patterns.


8:35 am on Oct 17, 2005 (gmt 0)

10+ Year Member

Come on folks, has nobody looked at this before?


9:11 am on Oct 18, 2005 (gmt 0)

10+ Year Member

Well, I don't know if that's of some value for you:

Gbot comes daily and requests robots.txt and root - at some busy sites he comes twice a day.

At least once every week or so he crawls deep through my sites visiting a lot of pages.

Constantly during those visits he tries to get some that doesn't exist:
GET /sortdir/asc HTTP/1.0
GET /eText/ HTTP/1.0
GET /eCat2/ HTTP/1.0
... always similar although 404ed.

New pages are best fed to him by deep links from home (better than from site map).

It takes max. 3 days for new pages to appear in the index after being crawled.


Featured Threads

Hot Threads This Week

Hot Threads This Month