Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Specific GoogleBot IPs spidering specific pages



1:01 pm on Feb 22, 2005 (gmt 0)

10+ Year Member

I haven't seen this before in this thread, so I thought I would bring this up. Has anyone else noticed specific GoogleBot IPs spidering specific pages on their websites? In pre-Allegra crawls, I never noticed any sort of pattern in how individual GoogleBots spidered my websites. But now, I am constantly noticing that the same GoogleBots hit the same pages (ie - hits page1.htm, page5.htm, page9.htm and hits page2.htm, page3.htm, page4.htm, and so on ... I have been watching this through very in depth log analysis since late december.

Is anyone else seeing this, and if so, are there any thoughts about why this pattern is occuring?


1:49 pm on Feb 23, 2005 (gmt 0)

10+ Year Member

Something else interesting to note ...

Specific GoogleBot IPs are retrieving pages as I had mentioned earlier, but also pages of specific word counts. What do I mean by this?

On one of my sites, I have a page for each product available, which has either a 1, 2 or 3 world page name (ie - widgets.htm, blue-widget.htm, blue-red-widget.htm), depending on the product. A certain GoogleBot IP has been spidering all 1 word named pages, another is spidering all 2 word named pages, and yet another is spidering all 3 word named pages. Always the same bots spidering the exact same pages every time ...


Featured Threads

Hot Threads This Week

Hot Threads This Month