Welcome to WebmasterWorld Guest from 54.163.49.19

Forum Moderators: phranque

Message Too Old, No Replies

Spider Software

     

fireice68

7:59 pm on Jan 2, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


Does anyone know of any good spider software other than Inktomi's that is out there. Thanks in advance.
2:37 am on Jan 3, 2001 (gmt 0)

Moderator

WebmasterWorld Administrator 10+ Year Member

joined:July 2, 2000
posts:2454
votes: 0


fireice68,

Welcome to Webmasterworld in 2001.

>>Does anyone know of any good spider software other than Inktomi's that is out there.

Were you looking for some software to spider various internet sites for you or were looking for a spider which crawls your own site every 24 hours like Inktomi's pay for listing? Once we kow what ou have in mind, I'm srue you will get plenty of help to find what you are looking for.

fireice68

5:38 pm on Jan 3, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


Software to spider various internet sites.
5:44 pm on Jan 3, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 7, 2000
posts:1720
votes: 0


Looking for poems or what?
7:31 pm on Jan 3, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


Fireice68 - the problem is we do not know what exactly you are looking for. There are lots of speciality spiders - some are web based and others are desktop apps. There is also licensed technology.

If you are interested in having a *small* search engine, for a type of speciality site check out FDSE [xav.com]. It is a high quality script.

fireice68

11:00 pm on Jan 3, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


Looking for specifically a web based technology.
11:15 pm on Jan 3, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


Check out [altavista.com ]
or [alltheweb.com ]

han solo

11:21 pm on Jan 3, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


In this, I think I am qualified to help.

[googlebot.com...]

[scrubtheweb.com...]

[excite.com...]

These all are spider based...googlebot works for google, excite.com has architext, and scrubtheweb.com has scrubby, although I've never seen it. Enjoy! :)

Cheers,
Han Solo

PS. Littleman, you mean altavista.com, or av.com, has a spider? Never seen it. ;)