homepage Welcome to WebmasterWorld Guest from 54.235.16.159
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Specific GoogleBot IPs spidering specific pages
catch2948




msg:727333
 1:01 pm on Feb 22, 2005 (gmt 0)

I haven't seen this before in this thread, so I thought I would bring this up. Has anyone else noticed specific GoogleBot IPs spidering specific pages on their websites? In pre-Allegra crawls, I never noticed any sort of pattern in how individual GoogleBots spidered my websites. But now, I am constantly noticing that the same GoogleBots hit the same pages (ie - 66.249.66.69 hits page1.htm, page5.htm, page9.htm and 66.249.64.38 hits page2.htm, page3.htm, page4.htm, and so on ... I have been watching this through very in depth log analysis since late december.

Is anyone else seeing this, and if so, are there any thoughts about why this pattern is occuring?

 

catch2948




msg:727334
 1:49 pm on Feb 23, 2005 (gmt 0)

Something else interesting to note ...

Specific GoogleBot IPs are retrieving pages as I had mentioned earlier, but also pages of specific word counts. What do I mean by this?

On one of my sites, I have a page for each product available, which has either a 1, 2 or 3 world page name (ie - widgets.htm, blue-widget.htm, blue-red-widget.htm), depending on the product. A certain GoogleBot IP has been spidering all 1 word named pages, another is spidering all 2 word named pages, and yet another is spidering all 3 word named pages. Always the same bots spidering the exact same pages every time ...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved