Welcome to WebmasterWorld Guest from 54.145.166.96

Message Too Old, No Replies

New FreshBot Behavior?

   
12:22 am on Feb 24, 2005 (gmt 0)

10+ Year Member



Just curious if anyone is seeing the following phenomenon:

A single Google Freshbot (66.249.66.43) spiders my site daily taking thousands of pages however, the the pages are spidered sequentially based on the number of words in the URI.

This particular bot is spidering the shorter URI's FIRST, then sequentially coming back for the longer mulit-phrase URI's. ie.

1-word-URI's
-then
2-word-URI's
-then
3-word-URI's
-then
4-word-URI's
etc.

Has anyone else seen this behavior?

 

Featured Threads

Hot Threads This Week

Hot Threads This Month