Welcome to WebmasterWorld Guest from 18.104.22.168
A single Google Freshbot (22.214.171.124) spiders my site daily taking thousands of pages however, the the pages are spidered based on the length of the URI (ie. [example.com...] where the keyword-phrase is the URI.
This particular bot is spidering the shorter URI's FIRST, then sequentially coming back for the longer mulit-phrase URI's. ie.
Has anyone else seen this behavior?
[edited by: ciml at 7:58 pm (utc) on Feb. 23, 2005]
[edit reason] Examplified [/edit]