homepage Welcome to WebmasterWorld Guest from 54.145.182.50
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
LightningDownload/1.0.1
Should this be labeled a good bot or a bad bot?
Pushycat

10+ Year Member



 
Msg#: 1328 posted 3:29 pm on Aug 15, 2002 (gmt 0)

The above is a product lets users select stuff to download from the web and a search turned up no mention of it here. The URL for the product is the same as the user agent name.

Should the fact it does not read robots.txt make it a bad bot that I exclude?

Or does the fact that the maximum requests it made in a 1 minute period were 22 with most of them at around 6 or 7 a minute make it a good bot even though it took stuff from folders that are excluded in robots.txt for all bots?

I'm trying to decide whether to ban it or not. Thanks in advance for your comments.

 

Knowles

10+ Year Member



 
Msg#: 1328 posted 3:37 pm on Aug 15, 2002 (gmt 0)

If you dont want it downloaded and taken then it would be bad. Most of these products dont have any real use other than pulling the website for either offline usage or to take the code out of it. I would ban it if it bothers you. If it doesnt dont worry about it.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved