homepage Welcome to WebmasterWorld Guest from 54.226.192.202
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
I dont know these
don_j

10+ Year Member



 
Msg#: 32 posted 12:22 pm on Feb 29, 2000 (gmt 0)

I've been looking at my logs and am pleased to find the old favourites are doing the rounds. I am however left with 2 questions and I hope you lot can help. Be gentle its my first time.

1. What is MSProxy , logikabot and testie ?
2. Scooter has crawled my site even though I've been having trouble with the 'Too many URLs' when submitting to Altavista. Can any one shed any light on the AltaVista situation.
3. Is Archie doing the rounds ? He's ignoring all my submissions.

Thanking you in hope of an answer.

Regards
Don

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 32 posted 3:18 am on Mar 1, 2000 (gmt 0)

I can't help with the MSProxy, logikabot and testie - I don't recognize those.

The fact that you have problems with AddUrl but are still getting crawled by scooter is refreshing. Are you sure it isn't just the "AddUrl scooter" version? Even when you are getting "too many urls" submitted, you will still get a visit from Scooter when you try to submit. Why it doesn't actually add the page is a mystery.

fantomaster

10+ Year Member



 
Msg#: 32 posted 5:39 am on Mar 1, 2000 (gmt 0)

Brett: Submission-crawling and not indexing after seems to be a generic technique characteristic
of the AV engine. We incurred a domain ban on an AV powered German engine last year due to involuntary
massive duplicate submission (human error ... ). Ever since, when I do a test submission, the spider
will hit the domain, there will even be a message saying that spidering was successful, but in the lower
part (different field) it will - only then - display the ban notice.

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 32 posted 3:20 am on Mar 2, 2000 (gmt 0)

I get the Add url spider, but never a crawl. They just snag the one page but I've not detected scooter on any banned domain (other than the 1 page grabbed during submission).

brett

fantomaster

10+ Year Member



 
Msg#: 32 posted 3:55 am on Mar 2, 2000 (gmt 0)

Yes, that's true. Sorry if I was unclear about it. What I'm basically saying is that when you resubmit a banned domain, they will still send their Add-Url spider obviously *in parallel* to checking their blacklist (possibly a probability thing to cut on response time: since we can reasonably assume that most submissions are valid, they won't let the spider wait till it gets confirmation from their banned domains database, or something).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved