homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Webmaster General
Forum Library, Charter, Moderators: phranque

Webmaster General Forum

Web Site Search Tools
What are the best tools to search a site
Fortune Hunter

 1:57 pm on Jun 10, 2008 (gmt 0)

I have a couple of web sites that are really starting to grow in terms of their content and number of pages. I need to implement a tool on the site to search and index the contents of the site and wondered what people thought the best tools to that were.

However one catch I have is I don't really want this tool to index some of my landing pages. I have pages that I use for promotions that are on the site, but I don't really want them indexed as "content" per se for obvious reasons.

Any suggestions of the best tools to do this would be appreciated.



 4:55 pm on Jun 10, 2008 (gmt 0)

I'd suggest a Google site search (simple enough to implement, just check your Google Webmaster Tools), along with a robots.txt that blocks the pages you don't want indexed.

Fortune Hunter

 1:58 pm on Jun 11, 2008 (gmt 0)

along with a robots.txt that blocks the pages you don't want indexed.

If I understand you correctly by including the robots.txt on the pages I don't want found the internal search will ignore those pages when it someone types in a search for something if it comes across that?


 3:36 pm on Jun 11, 2008 (gmt 0)

Google site search uses the Google index to find pages that it will use for your search results. If you block those pages from GoogleBot via robots.txt, they will not be in the Google index. So they will not show up in your site search results AND they will not show up when someone searches Google either.

I use Freefind for my site search (I can rebuild the site index any time I want), but I don't know if you can tell it not to add certain pages to the index.


 9:43 am on Jun 13, 2008 (gmt 0)

I have used Swish-e for years. It's free, fast and no db required. Not sure about the exclusion options, but the site should tell you what you need to know. They also have a mailing list.

One install on a server lets you use it on any number of sites by adding a config file and a PERL script for each. Runs off a CRON. Try swish-e (dot) org

Another that is supposed to be even faster us Swish++, but I've never tried it. It's on Sourceforge.


 7:13 pm on Jun 13, 2008 (gmt 0)

Is your site running off of a CMS (content management system) or is it just plain HTML pages? If your content grows large enough - I would recommend looking into developing your website on a CMS. Most all CMS systems have good search features built in. Then you don't have to rely on a service like Google.

Fortune Hunter

 9:57 pm on Jun 13, 2008 (gmt 0)

Thank you for the great tips. My site is currently just HTML pages, but I am looking at using a CMS for another site I am building for an unrelated business I am starting, which will have a lot more content and the need to be able to search it will be even more pronounced.


 1:14 pm on Jun 17, 2008 (gmt 0)

If i have understood you correctly, i use a php mysql prog called sphider.
It spiders and indexes all your pages content and stores in the DB.. You can then query DB with search strings and returns list of results (think this is done using MATCH AGAINST). I have found it really useful.


 1:59 pm on Jun 20, 2008 (gmt 0)

You can use Google Site Search, but why not include the Adsense one ? It's not a shame to make a few extra bucks from searches ;)

Fortune Hunter

 10:14 pm on Jun 20, 2008 (gmt 0)

but why not include the Adsense one ?

I know what Ad Sense is, but is the Ad Sense search tool? How does it work?


 6:16 am on Jun 21, 2008 (gmt 0)

The commercial version of Google site search costs about $100/year. That lets you show your chosen SERPs without AdSense. The free version of this tool requires that you show AdSense ads in the SERPs. I don't think that will be making you money.


 3:47 pm on Jun 21, 2008 (gmt 0)

We use the Adsense search box. Works quite well, though you do not get much control beyond robots.txt. Plus the extra income is nice.

At my last "real job" I installed a free search engine from Yahoo/IBM. Omni search or something like that. It was software based and worked quite well. And it was free.

Fortune Hunter

 1:48 am on Jun 24, 2008 (gmt 0)

Thank you for the suggestions. I will look into them, but the more I think about it the more I am thinking for this new project I will need a good CMS system which will have a built in search function.


 2:00 pm on Jun 24, 2008 (gmt 0)

Sounds like a CMS is your ideal solution.

In the meantime, the suggestions about Goog site search are good. You should also take a peak at the open source PHP script called 'Sphider'.

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Webmaster General
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved