Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies


Should this be labeled a good bot or a bad bot?

3:29 pm on Aug 15, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 21, 2002
votes: 0

The above is a product lets users select stuff to download from the web and a search turned up no mention of it here. The URL for the product is the same as the user agent name.

Should the fact it does not read robots.txt make it a bad bot that I exclude?

Or does the fact that the maximum requests it made in a 1 minute period were 22 with most of them at around 6 or 7 a minute make it a good bot even though it took stuff from folders that are excluded in robots.txt for all bots?

I'm trying to decide whether to ban it or not. Thanks in advance for your comments.

3:37 pm on Aug 15, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 17, 2002
votes: 0

If you dont want it downloaded and taken then it would be bad. Most of these products dont have any real use other than pulling the website for either offline usage or to take the code out of it. I would ban it if it bothers you. If it doesnt dont worry about it.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members