Welcome to WebmasterWorld Guest from 54.226.133.245

Forum Moderators: phranque

Message Too Old, No Replies

Spidering Search Engine ?

I need one!

     
6:58 pm on Jul 8, 2001 (gmt 0)

Full Member

10+ Year Member

joined:July 8, 2001
posts:267
votes: 0


Are there any commercial search scripts that spider URLs that I supply? I need a search script that indexes a site based on its <body>

Most other search scripts i tried just search by the description the person posted with their URL submission... I dont want to keep updating the description for the URL everytime that page changes.

In summary i need a (CGI/PERL) script that spiders pages every so often. I just need it to spider about a dozen.

Ive tried the one at [xav.com...] but it only seems to spider the pages on my own host ....

Can someone pleeeeaaaassssseeeee help :)

Thanks a lot!

11:23 pm on July 8, 2001 (gmt 0)

Senior Member

joined:June 27, 2000
posts:1548
votes: 0


We're usually a little quicker than this, KingKelly. I am kicking this up to the top for some help...

-G

3:08 am on July 9, 2001 (gmt 0)

Full Member

10+ Year Member

joined:July 8, 2001
posts:267
votes: 0


Thanks a bunch ;)

Friendly forum you got here

7:45 am on July 9, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 1, 2001
posts:183
votes: 0


Welcome to WebmasterWorld kingkelly

Maybe you should try posting a question to the Fluid Dynamics Discussion Board [xav.com] because that search script should spider other urls.

If I'm not mistaken, that is the same search engine script that this site uses and Brett spiders this site as well as Search Engine World [searchengineworld.com] with it.

I use it myself for our site but that is the only one we ever spidered.

5:14 pm on July 9, 2001 (gmt 0)

Full Member

10+ Year Member

joined:July 8, 2001
posts:267
votes: 0


Hmmmm....

Perhaps its because I tried it on a free hosting site? [free.prohosting.com...]

I like their services but perhaps they dont have a certain module installed???

Who knows... Im just some 16 year old kid who needs a spidreing script to beef up his resume..... (gettin tired of McDonalds)

If anyone knows of another script or free host that supports the Fluid Dynamics script than please email or post it in this forum ;) ;) ;)

5:38 pm on July 9, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


kingkelly, it could be that they are restricting socket connections. Did you try the auto install? Maybe try the free hosting at hypermart. I have had good luck running spiders off of their free sites.

If you are doing this stuff at 16, you won't be working at McDonald's long.

6:04 pm on July 9, 2001 (gmt 0)

Full Member

10+ Year Member

joined:July 8, 2001
posts:267
votes: 0


tee-hee...

Actually i jsut checked fluid dynamics website, and it said that it definetly works on hypermart (but now i have to put in a bllody credit card #)... I have a few fake ones, but im worried that they automatically charge me if i exceed the bandwidth limit...

Im assuming the spidering search script would use up a lot of bandwidth? And i did try the auto install.. Im trying it now on netfirms..

It takesa like a minute or 2.... I tried testing a spider search on google.com, and it said error:404 not found. I tried it on some other minor sites and got the same. When i had it spider netfirms.com, there was no prob bob.... According to their help it said the script will partially work on netfirms and prohosting since they ahve limited sockets installed....
--->"CGI supported, but without full sockets privileges -- FDSE search engine won't be fully functional"

So, prohosting and netfirms only spider their own server, and im too scared to try hypermart.... :(

What is a boy to do?

6:36 pm on July 9, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 10, 2000
posts:2151
votes: 0


KK - if you do the free thing at hypermart they shouldn't ask for a credit card. I setup one just last week with all fake info and no problems and not cc needed.
7:09 pm on July 9, 2001 (gmt 0)

Full Member

10+ Year Member

joined:July 8, 2001
posts:267
votes: 0


In step 3 of 3, it asks for credit card information... if i leave it blank it wont let me continue....
4:36 am on July 10, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
posts:12095
votes: 0


Something's changed, oilman. I just tried signing up - there are still free accounts avaiable, but I could not complete the application without giving credit card information.

They seem to be requiring it in case the account goes over disk space or bandwidth allotments.

4:49 am on July 10, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


What a pain.