Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
I haven't seen this before, but then I haven't looked in a while. Did AV unleash a freshbot? It's not appearing on all sites.
Incidentally, I never registered any of these pages in AV's PFI program.
Anyone can inform me how often there will be an update? Or do they add links continuesly, and if so, after how many days after the Scooter came by the links will be visible?
Just today I noticed a 'referrer' or something from Scooter like this:
126.96.36.199 - - [15/Apr/2003:01:05:39 -0700] "GET /robots.txt HTTP/1.1" 200 2256 "htt*://www.some-site-that-has-nothing-to-do-with-mine/page.htm" "Scooter/3.3_SF"
Any ideas here?
Kidding aside, though, making spidering frequency dependent on click rate does seem like a sensible way to keep from wasting spidering and processing bandwidth on pages that no one cares about.