I've been looking at my logs and am pleased to find the old favourites are doing the rounds. I am however left with 2 questions and I hope you lot can help. Be gentle its my first time.
1. What is MSProxy , logikabot and testie ? 2. Scooter has crawled my site even though I've been having trouble with the 'Too many URLs' when submitting to Altavista. Can any one shed any light on the AltaVista situation. 3. Is Archie doing the rounds ? He's ignoring all my submissions.
I can't help with the MSProxy, logikabot and testie - I don't recognize those.
The fact that you have problems with AddUrl but are still getting crawled by scooter is refreshing. Are you sure it isn't just the "AddUrl scooter" version? Even when you are getting "too many urls" submitted, you will still get a visit from Scooter when you try to submit. Why it doesn't actually add the page is a mystery.
Brett: Submission-crawling and not indexing after seems to be a generic technique characteristic of the AV engine. We incurred a domain ban on an AV powered German engine last year due to involuntary massive duplicate submission (human error ... ). Ever since, when I do a test submission, the spider will hit the domain, there will even be a message saying that spidering was successful, but in the lower part (different field) it will - only then - display the ban notice.
Yes, that's true. Sorry if I was unclear about it. What I'm basically saying is that when you resubmit a banned domain, they will still send their Add-Url spider obviously *in parallel* to checking their blacklist (possibly a probability thing to cut on response time: since we can reasonably assume that most submissions are valid, they won't let the spider wait till it gets confirmation from their banned domains database, or something).