URL Spider Pro

[misosoph]

66.218.239.13 - - [10/May/2002:10:26:28 -0400] "GET /robots.txt HTTP/1.0" 200 2290 "-" "URL Spider Pro/1.92 (innerprise.net)"

The IP belongs to CableAmerica Corporation (Mesa, AZ, US).

This requested only robots.txt and homepage.

There is a description of the robot here:
http://www.innerprise.net/usp-index.asp

Is this friend or foe?

[volatilegx]

If it's a publicly distributed bot, then I'd say it's a foe, except in special cases.

[misosoph]

Thank you.

Is this "URL Spider Pro" that I found in my access logs today the kind of information that I ought to report to this forum? This belongs to this forum's subject matter, doesn't it?

[Woz]

URL Spider Pro is a commercially available spider program for building URL Search Engines (Duh). I have played with it and it seems quite good at collecting data although the paramater setting is a little limited. Being moderately priced it is aimed at people setting up smaller engines or niche directories although there is an SQLserver version also available for larger installations.

So it could be a friend or a foe, depending on who is using it and why.

In these days of diminishing free submissions I would be looking closer at the IP rather than the product. I think it would be important to determine whether the usage is to build a legitimate Engine which could result in referrals or a home camper simply playing. The former would be preferable, the latter bannable (is that a word??).

Again a case of Babies and Bathwater...

Onya
Woz

[saurabh]

What is "a case of Babies and Bathwater"?

[Woz]

It's my reference to an old saying "Don't throw the baby out with the bathwater".

When one has finished washing the baby, one throws out the water. One must however remove the baby from the water first to avoid throwing out the baby with the bathwater.

In this case, by banning URL Spider Pro you may be losing out on a potential listing in a directory and the resultant referrals.

Onya
Woz

[jdMorgan]

saurabh,

What is "a case of Babies and Bathwater"?

Making the mistake of throwing out the baby along with the water you just bathed it in.
Discarding something of high value along with waste.

To put it into context, Woz is arguing that the URL Spider Pro should not be banned by User-agent because it can be used to build a legitimate search index that may send traffic to your site. Because the number of search engines that will list your site without a fee is diminishing, small independent search engines built using programs like URL Spider Pro may become valuable to smaller, low-budget Web sites in the future.

Listing URL Spider Pro here in this forum makes it likely that many Webmasters will ban it.

Woz is urging caution, because banning it by User-agent may be throwing away some future site traffic to avoid a few who misuse URL Spider Pro for "bad purposes" like e-mail address harvesting. Instead, Woz suggests banning the IP addresses of those who use URL Spider Pro to abuse your site.

Although my intent was to clarify the "bathwater" saying, that's how I read what Woz had to say, and I'll welcome a correction if I got it wrong!

Jim

<edit>Woz beat me to it!</edit>