homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Specific GoogleBot IPs spidering specific pages

 1:01 pm on Feb 22, 2005 (gmt 0)

I haven't seen this before in this thread, so I thought I would bring this up. Has anyone else noticed specific GoogleBot IPs spidering specific pages on their websites? In pre-Allegra crawls, I never noticed any sort of pattern in how individual GoogleBots spidered my websites. But now, I am constantly noticing that the same GoogleBots hit the same pages (ie - hits page1.htm, page5.htm, page9.htm and hits page2.htm, page3.htm, page4.htm, and so on ... I have been watching this through very in depth log analysis since late december.

Is anyone else seeing this, and if so, are there any thoughts about why this pattern is occuring?



 1:49 pm on Feb 23, 2005 (gmt 0)

Something else interesting to note ...

Specific GoogleBot IPs are retrieving pages as I had mentioned earlier, but also pages of specific word counts. What do I mean by this?

On one of my sites, I have a page for each product available, which has either a 1, 2 or 3 world page name (ie - widgets.htm, blue-widget.htm, blue-red-widget.htm), depending on the product. A certain GoogleBot IP has been spidering all 1 word named pages, another is spidering all 2 word named pages, and yet another is spidering all 3 word named pages. Always the same bots spidering the exact same pages every time ...

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved