homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

I dont know these

 12:22 pm on Feb 29, 2000 (gmt 0)

I've been looking at my logs and am pleased to find the old favourites are doing the rounds. I am however left with 2 questions and I hope you lot can help. Be gentle its my first time.

1. What is MSProxy , logikabot and testie ?
2. Scooter has crawled my site even though I've been having trouble with the 'Too many URLs' when submitting to Altavista. Can any one shed any light on the AltaVista situation.
3. Is Archie doing the rounds ? He's ignoring all my submissions.

Thanking you in hope of an answer.




 3:18 am on Mar 1, 2000 (gmt 0)

I can't help with the MSProxy, logikabot and testie - I don't recognize those.

The fact that you have problems with AddUrl but are still getting crawled by scooter is refreshing. Are you sure it isn't just the "AddUrl scooter" version? Even when you are getting "too many urls" submitted, you will still get a visit from Scooter when you try to submit. Why it doesn't actually add the page is a mystery.


 5:39 am on Mar 1, 2000 (gmt 0)

Brett: Submission-crawling and not indexing after seems to be a generic technique characteristic
of the AV engine. We incurred a domain ban on an AV powered German engine last year due to involuntary
massive duplicate submission (human error ... ). Ever since, when I do a test submission, the spider
will hit the domain, there will even be a message saying that spidering was successful, but in the lower
part (different field) it will - only then - display the ban notice.


 3:20 am on Mar 2, 2000 (gmt 0)

I get the Add url spider, but never a crawl. They just snag the one page but I've not detected scooter on any banned domain (other than the 1 page grabbed during submission).



 3:55 am on Mar 2, 2000 (gmt 0)

Yes, that's true. Sorry if I was unclear about it. What I'm basically saying is that when you resubmit a banned domain, they will still send their Add-Url spider obviously *in parallel* to checking their blacklist (possibly a probability thing to cut on response time: since we can reasonably assume that most submissions are valid, they won't let the spider wait till it gets confirmation from their banned domains database, or something).

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved