Welcome to WebmasterWorld Guest from 54.146.240.181

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

URL Spider Pro

     

misosoph

5:11 pm on May 10, 2002 (gmt 0)

10+ Year Member



66.218.239.13 - - [10/May/2002:10:26:28 -0400] "GET /robots.txt HTTP/1.0" 200 2290 "-" "URL Spider Pro/1.92 (innerprise.net)"

The IP belongs to CableAmerica Corporation (Mesa, AZ, US).

This requested only robots.txt and homepage.

There is a description of the robot here:
http://www.innerprise.net/usp-index.asp

Is this friend or foe?

volatilegx

6:08 pm on May 10, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If it's a publicly distributed bot, then I'd say it's a foe, except in special cases.

misosoph

4:52 am on May 11, 2002 (gmt 0)

10+ Year Member



Thank you.

Is this "URL Spider Pro" that I found in my access logs today the kind of information that I ought to report to this forum? This belongs to this forum's subject matter, doesn't it?

Woz

5:18 am on May 11, 2002 (gmt 0)

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member



URL Spider Pro is a commercially available spider program for building URL Search Engines (Duh). I have played with it and it seems quite good at collecting data although the paramater setting is a little limited. Being moderately priced it is aimed at people setting up smaller engines or niche directories although there is an SQLserver version also available for larger installations.

So it could be a friend or a foe, depending on who is using it and why.

In these days of diminishing free submissions I would be looking closer at the IP rather than the product. I think it would be important to determine whether the usage is to build a legitimate Engine which could result in referrals or a home camper simply playing. The former would be preferable, the latter bannable (is that a word??).

Again a case of Babies and Bathwater...

Onya
Woz

saurabh

10:31 pm on Oct 13, 2002 (gmt 0)

10+ Year Member



What is "a case of Babies and Bathwater"?

Woz

10:48 pm on Oct 13, 2002 (gmt 0)

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member



It's my reference to an old saying "Don't throw the baby out with the bathwater".

When one has finished washing the baby, one throws out the water. One must however remove the baby from the water first to avoid throwing out the baby with the bathwater.

In this case, by banning URL Spider Pro you may be losing out on a potential listing in a directory and the resultant referrals.

Onya
Woz

jdMorgan

10:58 pm on Oct 13, 2002 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



saurabh,

What is "a case of Babies and Bathwater"?

Making the mistake of throwing out the baby along with the water you just bathed it in.
Discarding something of high value along with waste.

To put it into context, Woz is arguing that the URL Spider Pro should not be banned by User-agent because it can be used to build a legitimate search index that may send traffic to your site. Because the number of search engines that will list your site without a fee is diminishing, small independent search engines built using programs like URL Spider Pro may become valuable to smaller, low-budget Web sites in the future.

Listing URL Spider Pro here in this forum makes it likely that many Webmasters will ban it.

Woz is urging caution, because banning it by User-agent may be throwing away some future site traffic to avoid a few who misuse URL Spider Pro for "bad purposes" like e-mail address harvesting. Instead, Woz suggests banning the IP addresses of those who use URL Spider Pro to abuse your site.

Although my intent was to clarify the "bathwater" saying, that's how I read what Woz had to say, and I'll welcome a correction if I got it wrong!

Jim

<edit>Woz beat me to it!</edit>

 

Featured Threads

Hot Threads This Week

Hot Threads This Month