Welcome to WebmasterWorld Guest from 54.167.83.224

Message Too Old, No Replies

Googlebot - Sequentially Spidering?

     
6:58 pm on Feb 23, 2005 (gmt 0)

New User

10+ Year Member

joined:Feb 23, 2005
posts:6
votes: 0


Just curious if anyone is seeing the following phenomenon:

A single Google Freshbot (66.249.66.43) spiders my site daily taking thousands of pages however, the the pages are spidered based on the length of the URI (ie. [example.com...] where the keyword-phrase is the URI.

This particular bot is spidering the shorter URI's FIRST, then sequentially coming back for the longer mulit-phrase URI's. ie.

[example.com...]
[example.com...]
[example.com...]
[example.com...]
etc.

Has anyone else seen this behavior?

[edited by: ciml at 7:58 pm (utc) on Feb. 23, 2005]
[edit reason] Examplified [/edit]

8:05 pm on Feb 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 25, 2002
posts:153
votes: 0


yeah, i've seen that sometimes ... i think there have been some threads discussing this before.