homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

What can I learn from Googlebot activity in site logs?

10+ Year Member

Msg#: 31641 posted 5:00 pm on Oct 14, 2005 (gmt 0)


I've been looking at some of my site logs recently and wondering whether there's anything I can glean from them other than "oh, Googlebot's picked up my new pages" or "Wow, Googlebot's requesting a lot of pages today". Is it possible to infer anything from frequency or order that G accesses the pages? Repeated requests for a single page? Does the request order tell me anything about my link structure? I presume G works on some sort of logical basis, hence my query.

Apologies in advance if this is a noob subject...!

All comments welcome.



WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 31641 posted 5:59 am on Oct 15, 2005 (gmt 0)

Shameless bump here -- I'd also be very interested in any conclusions people have drawn from Googlebot patterns.


10+ Year Member

Msg#: 31641 posted 8:35 am on Oct 17, 2005 (gmt 0)

Come on folks, has nobody looked at this before?


10+ Year Member

Msg#: 31641 posted 9:11 am on Oct 18, 2005 (gmt 0)

Well, I don't know if that's of some value for you:

Gbot comes daily and requests robots.txt and root - at some busy sites he comes twice a day.

At least once every week or so he crawls deep through my sites visiting a lot of pages.

Constantly during those visits he tries to get some that doesn't exist:
GET /sortdir/asc HTTP/1.0
GET /eText/ HTTP/1.0
GET /eCat2/ HTTP/1.0
... always similar although 404ed.

New pages are best fed to him by deep links from home (better than from site map).

It takes max. 3 days for new pages to appear in the index after being crawled.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved