|Specific GoogleBot IPs spidering specific pages|
I haven't seen this before in this thread, so I thought I would bring this up. Has anyone else noticed specific GoogleBot IPs spidering specific pages on their websites? In pre-Allegra crawls, I never noticed any sort of pattern in how individual GoogleBots spidered my websites. But now, I am constantly noticing that the same GoogleBots hit the same pages (ie - 220.127.116.11 hits page1.htm, page5.htm, page9.htm and 18.104.22.168 hits page2.htm, page3.htm, page4.htm, and so on ... I have been watching this through very in depth log analysis since late december.
Is anyone else seeing this, and if so, are there any thoughts about why this pattern is occuring?
Something else interesting to note ...
Specific GoogleBot IPs are retrieving pages as I had mentioned earlier, but also pages of specific word counts. What do I mean by this?
On one of my sites, I have a page for each product available, which has either a 1, 2 or 3 world page name (ie - widgets.htm, blue-widget.htm, blue-red-widget.htm), depending on the product. A certain GoogleBot IP has been spidering all 1 word named pages, another is spidering all 2 word named pages, and yet another is spidering all 3 word named pages. Always the same bots spidering the exact same pages every time ...