homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

Spidering Search Engine ?
I need one!

 6:58 pm on Jul 8, 2001 (gmt 0)

Are there any commercial search scripts that spider URLs that I supply? I need a search script that indexes a site based on its <body>

Most other search scripts i tried just search by the description the person posted with their URL submission... I dont want to keep updating the description for the URL everytime that page changes.

In summary i need a (CGI/PERL) script that spiders pages every so often. I just need it to spider about a dozen.

Ive tried the one at [xav.com...] but it only seems to spider the pages on my own host ....

Can someone pleeeeaaaassssseeeee help :)

Thanks a lot!



 11:23 pm on Jul 8, 2001 (gmt 0)

We're usually a little quicker than this, KingKelly. I am kicking this up to the top for some help...



 3:08 am on Jul 9, 2001 (gmt 0)

Thanks a bunch ;)

Friendly forum you got here


 7:45 am on Jul 9, 2001 (gmt 0)

Welcome to WebmasterWorld kingkelly

Maybe you should try posting a question to the Fluid Dynamics Discussion Board [xav.com] because that search script should spider other urls.

If I'm not mistaken, that is the same search engine script that this site uses and Brett spiders this site as well as Search Engine World [searchengineworld.com] with it.

I use it myself for our site but that is the only one we ever spidered.


 5:14 pm on Jul 9, 2001 (gmt 0)


Perhaps its because I tried it on a free hosting site? [free.prohosting.com...]

I like their services but perhaps they dont have a certain module installed???

Who knows... Im just some 16 year old kid who needs a spidreing script to beef up his resume..... (gettin tired of McDonalds)

If anyone knows of another script or free host that supports the Fluid Dynamics script than please email or post it in this forum ;) ;) ;)


 5:38 pm on Jul 9, 2001 (gmt 0)

kingkelly, it could be that they are restricting socket connections. Did you try the auto install? Maybe try the free hosting at hypermart. I have had good luck running spiders off of their free sites.

If you are doing this stuff at 16, you won't be working at McDonald's long.


 6:04 pm on Jul 9, 2001 (gmt 0)


Actually i jsut checked fluid dynamics website, and it said that it definetly works on hypermart (but now i have to put in a bllody credit card #)... I have a few fake ones, but im worried that they automatically charge me if i exceed the bandwidth limit...

Im assuming the spidering search script would use up a lot of bandwidth? And i did try the auto install.. Im trying it now on netfirms..

It takesa like a minute or 2.... I tried testing a spider search on google.com, and it said error:404 not found. I tried it on some other minor sites and got the same. When i had it spider netfirms.com, there was no prob bob.... According to their help it said the script will partially work on netfirms and prohosting since they ahve limited sockets installed....
--->"CGI supported, but without full sockets privileges -- FDSE search engine won't be fully functional"

So, prohosting and netfirms only spider their own server, and im too scared to try hypermart.... :(

What is a boy to do?


 6:36 pm on Jul 9, 2001 (gmt 0)

KK - if you do the free thing at hypermart they shouldn't ask for a credit card. I setup one just last week with all fake info and no problems and not cc needed.


 7:09 pm on Jul 9, 2001 (gmt 0)

In step 3 of 3, it asks for credit card information... if i leave it blank it wont let me continue....


 4:36 am on Jul 10, 2001 (gmt 0)

Something's changed, oilman. I just tried signing up - there are still free accounts avaiable, but I could not complete the application without giving credit card information.

They seem to be requiring it in case the account goes over disk space or bandwidth allotments.


 4:49 am on Jul 10, 2001 (gmt 0)

What a pain.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved