homepage Welcome to WebmasterWorld Guest from 54.204.215.209
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

    
Spidering Search Engine ?
I need one!
kingkelly




msg:673846
 6:58 pm on Jul 8, 2001 (gmt 0)

Are there any commercial search scripts that spider URLs that I supply? I need a search script that indexes a site based on its <body>

Most other search scripts i tried just search by the description the person posted with their URL submission... I dont want to keep updating the description for the URL everytime that page changes.

In summary i need a (CGI/PERL) script that spiders pages every so often. I just need it to spider about a dozen.

Ive tried the one at [xav.com...] but it only seems to spider the pages on my own host ....

Can someone pleeeeaaaassssseeeee help :)

Thanks a lot!

 

grnidone




msg:673847
 11:23 pm on Jul 8, 2001 (gmt 0)

We're usually a little quicker than this, KingKelly. I am kicking this up to the top for some help...

-G

kingkelly




msg:673848
 3:08 am on Jul 9, 2001 (gmt 0)

Thanks a bunch ;)

Friendly forum you got here

Son_House




msg:673849
 7:45 am on Jul 9, 2001 (gmt 0)

Welcome to WebmasterWorld kingkelly

Maybe you should try posting a question to the Fluid Dynamics Discussion Board [xav.com] because that search script should spider other urls.

If I'm not mistaken, that is the same search engine script that this site uses and Brett spiders this site as well as Search Engine World [searchengineworld.com] with it.

I use it myself for our site but that is the only one we ever spidered.

kingkelly




msg:673850
 5:14 pm on Jul 9, 2001 (gmt 0)

Hmmmm....

Perhaps its because I tried it on a free hosting site? [free.prohosting.com...]

I like their services but perhaps they dont have a certain module installed???

Who knows... Im just some 16 year old kid who needs a spidreing script to beef up his resume..... (gettin tired of McDonalds)

If anyone knows of another script or free host that supports the Fluid Dynamics script than please email or post it in this forum ;) ;) ;)

littleman




msg:673851
 5:38 pm on Jul 9, 2001 (gmt 0)

kingkelly, it could be that they are restricting socket connections. Did you try the auto install? Maybe try the free hosting at hypermart. I have had good luck running spiders off of their free sites.

If you are doing this stuff at 16, you won't be working at McDonald's long.

kingkelly




msg:673852
 6:04 pm on Jul 9, 2001 (gmt 0)

tee-hee...

Actually i jsut checked fluid dynamics website, and it said that it definetly works on hypermart (but now i have to put in a bllody credit card #)... I have a few fake ones, but im worried that they automatically charge me if i exceed the bandwidth limit...

Im assuming the spidering search script would use up a lot of bandwidth? And i did try the auto install.. Im trying it now on netfirms..

It takesa like a minute or 2.... I tried testing a spider search on google.com, and it said error:404 not found. I tried it on some other minor sites and got the same. When i had it spider netfirms.com, there was no prob bob.... According to their help it said the script will partially work on netfirms and prohosting since they ahve limited sockets installed....
--->"CGI supported, but without full sockets privileges -- FDSE search engine won't be fully functional"

So, prohosting and netfirms only spider their own server, and im too scared to try hypermart.... :(

What is a boy to do?

oilman




msg:673853
 6:36 pm on Jul 9, 2001 (gmt 0)

KK - if you do the free thing at hypermart they shouldn't ask for a credit card. I setup one just last week with all fake info and no problems and not cc needed.

kingkelly




msg:673854
 7:09 pm on Jul 9, 2001 (gmt 0)

In step 3 of 3, it asks for credit card information... if i leave it blank it wont let me continue....

Marcia




msg:673855
 4:36 am on Jul 10, 2001 (gmt 0)

Something's changed, oilman. I just tried signing up - there are still free accounts avaiable, but I could not complete the application without giving credit card information.

They seem to be requiring it in case the account goes over disk space or bandwidth allotments.

littleman




msg:673856
 4:49 am on Jul 10, 2001 (gmt 0)

What a pain.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved