homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

URL Spider Pro

 5:11 pm on May 10, 2002 (gmt 0) - - [10/May/2002:10:26:28 -0400] "GET /robots.txt HTTP/1.0" 200 2290 "-" "URL Spider Pro/1.92 (innerprise.net)"

The IP belongs to CableAmerica Corporation (Mesa, AZ, US).

This requested only robots.txt and homepage.

There is a description of the robot here:

Is this friend or foe?



 6:08 pm on May 10, 2002 (gmt 0)

If it's a publicly distributed bot, then I'd say it's a foe, except in special cases.


 4:52 am on May 11, 2002 (gmt 0)

Thank you.

Is this "URL Spider Pro" that I found in my access logs today the kind of information that I ought to report to this forum? This belongs to this forum's subject matter, doesn't it?


 5:18 am on May 11, 2002 (gmt 0)

URL Spider Pro is a commercially available spider program for building URL Search Engines (Duh). I have played with it and it seems quite good at collecting data although the paramater setting is a little limited. Being moderately priced it is aimed at people setting up smaller engines or niche directories although there is an SQLserver version also available for larger installations.

So it could be a friend or a foe, depending on who is using it and why.

In these days of diminishing free submissions I would be looking closer at the IP rather than the product. I think it would be important to determine whether the usage is to build a legitimate Engine which could result in referrals or a home camper simply playing. The former would be preferable, the latter bannable (is that a word??).

Again a case of Babies and Bathwater...



 10:31 pm on Oct 13, 2002 (gmt 0)

What is "a case of Babies and Bathwater"?


 10:48 pm on Oct 13, 2002 (gmt 0)

It's my reference to an old saying "Don't throw the baby out with the bathwater".

When one has finished washing the baby, one throws out the water. One must however remove the baby from the water first to avoid throwing out the baby with the bathwater.

In this case, by banning URL Spider Pro you may be losing out on a potential listing in a directory and the resultant referrals.



 10:58 pm on Oct 13, 2002 (gmt 0)


What is "a case of Babies and Bathwater"?

Making the mistake of throwing out the baby along with the water you just bathed it in.
Discarding something of high value along with waste.

To put it into context, Woz is arguing that the URL Spider Pro should not be banned by User-agent because it can be used to build a legitimate search index that may send traffic to your site. Because the number of search engines that will list your site without a fee is diminishing, small independent search engines built using programs like URL Spider Pro may become valuable to smaller, low-budget Web sites in the future.

Listing URL Spider Pro here in this forum makes it likely that many Webmasters will ban it.

Woz is urging caution, because banning it by User-agent may be throwing away some future site traffic to avoid a few who misuse URL Spider Pro for "bad purposes" like e-mail address harvesting. Instead, Woz suggests banning the IP addresses of those who use URL Spider Pro to abuse your site.

Although my intent was to clarify the "bathwater" saying, that's how I read what Woz had to say, and I'll welcome a correction if I got it wrong!


<edit>Woz beat me to it!</edit>

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved