Forum Moderators: open
[webmasterworld.com...]
If you have followed how Mercator spiders your site it pretty much matches what is described in this bit of info.
IMHO, Mercator has nothing to do at this moment in how Altavista is indexing their current results. They are probably testing out some new spidering techniques and sorting methods they want to implement in the future.
209.73.164.126 - - [17/Feb/2001:02:15:52 -0500] "GET /robots.txt HTTP/1.0" 404 204 "-" "Mercator-Scrub-1.1"
209.73.164.126 - - [17/Feb/2001:02:15:52 -0500] "GET / HTTP/1.0" 200 4285 "-" "Mercator-Scrub-1.1"
Also, at a site that's been in for a while, same one showed up - only one remote inside page, totally random:
209.73.164.126 - - [17/Feb/2001:09:18:24 -0500] "GET /robots.txt HTTP/1.0" 404 204 "-" "Mercator-Scrub-1.1"
209.73.164.126 - - [17/Feb/2001:09:18:24 -0500] "GET /Graphics/Tiles/corduroy/ HTTP/1.0" 200 5329 "-" "Mercator-Scrub-1.1"
I generally don't pay close attention, but I've not seen this one before, and this struck me as odd - just the one odd page.
I'm guessing AV parked Scooter for a couple of months, either to tune it and/or in anticipation of LookSmart. Now it appears all those long lost submitted URLs are now being requested and indexed
I keep very detailed logs of all submissions as well as detailed logs of all site activity. After reviewing all logs etc, I can safely say (93.2% certain Regis), that AV is processing a backlog of URL submissions since early December.
The recent index update on AV contains pages that were submitted DEC-4 thru DEC-6 (using a web multi-submission tool). Scooter (kalyan) accessed/indexed those pages (other then the instant verify), on FEB-1 and FEB-2. These pages are now in the current index. I have other URLs in the queue so I should be able to follow up with this theory. The next by date submission was JAN-7.