Web Site Search Tools

What are the best tools to search a site

[Fortune Hunter]

I have a couple of web sites that are really starting to grow in terms of their content and number of pages. I need to implement a tool on the site to search and index the contents of the site and wondered what people thought the best tools to that were.

However one catch I have is I don't really want this tool to index some of my landing pages. I have pages that I use for promotions that are on the site, but I don't really want them indexed as "content" per se for obvious reasons.

Any suggestions of the best tools to do this would be appreciated.

[WesleyC]

I'd suggest a Google site search (simple enough to implement, just check your Google Webmaster Tools), along with a robots.txt that blocks the pages you don't want indexed.

[Fortune Hunter]

along with a robots.txt that blocks the pages you don't want indexed.

If I understand you correctly by including the robots.txt on the pages I don't want found the internal search will ignore those pages when it someone types in a search for something if it comes across that?

[wrgvt]

Google site search uses the Google index to find pages that it will use for your search results. If you block those pages from GoogleBot via robots.txt, they will not be in the Google index. So they will not show up in your site search results AND they will not show up when someone searches Google either.

I use Freefind for my site search (I can rebuild the site index any time I want), but I don't know if you can tell it not to add certain pages to the index.

[old_expat]

I have used Swish-e for years. It's free, fast and no db required. Not sure about the exclusion options, but the site should tell you what you need to know. They also have a mailing list.

One install on a server lets you use it on any number of sites by adding a config file and a PERL script for each. Runs off a CRON. Try swish-e (dot) org

Another that is supposed to be even faster us Swish++, but I've never tried it. It's on Sourceforge.

[naitsirhc26]

Is your site running off of a CMS (content management system) or is it just plain HTML pages? If your content grows large enough - I would recommend looking into developing your website on a CMS. Most all CMS systems have good search features built in. Then you don't have to rely on a service like Google.

[Fortune Hunter]

Thank you for the great tips. My site is currently just HTML pages, but I am looking at using a CMS for another site I am building for an unrelated business I am starting, which will have a lot more content and the need to be able to search it will be even more pronounced.

[Scally_Ally]

If i have understood you correctly, i use a php mysql prog called sphider.
It spiders and indexes all your pages content and stores in the DB.. You can then query DB with search strings and returns list of results (think this is done using MATCH AGAINST). I have found it really useful.
Ally

[LaurentB]

You can use Google Site Search, but why not include the Adsense one ? It's not a shame to make a few extra bucks from searches ;)

[Fortune Hunter]

but why not include the Adsense one ?

I know what Ad Sense is, but is the Ad Sense search tool? How does it work?

[bill]

The commercial version of Google site search costs about $100/year. That lets you show your chosen SERPs without AdSense. The free version of this tool requires that you show AdSense ads in the SERPs. I don't think that will be making you money.

[maximillianos]

We use the Adsense search box. Works quite well, though you do not get much control beyond robots.txt. Plus the extra income is nice.

At my last "real job" I installed a free search engine from Yahoo/IBM. Omni search or something like that. It was software based and worked quite well. And it was free.

[Fortune Hunter]

Thank you for the suggestions. I will look into them, but the more I think about it the more I am thinking for this new project I will need a good CMS system which will have a built in search function.

[justgowithit]

Sounds like a CMS is your ideal solution.

In the meantime, the suggestions about Goog site search are good. You should also take a peak at the open source PHP script called 'Sphider'.