Welcome to WebmasterWorld Guest from 54.162.239.134

Forum Moderators: phranque

Message Too Old, No Replies

Web Site Search Tools

What are the best tools to search a site

     
1:57 pm on Jun 10, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have a couple of web sites that are really starting to grow in terms of their content and number of pages. I need to implement a tool on the site to search and index the contents of the site and wondered what people thought the best tools to that were.

However one catch I have is I don't really want this tool to index some of my landing pages. I have pages that I use for promotions that are on the site, but I don't really want them indexed as "content" per se for obvious reasons.

Any suggestions of the best tools to do this would be appreciated.

4:55 pm on Jun 10, 2008 (gmt 0)

5+ Year Member



I'd suggest a Google site search (simple enough to implement, just check your Google Webmaster Tools), along with a robots.txt that blocks the pages you don't want indexed.
1:58 pm on Jun 11, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



along with a robots.txt that blocks the pages you don't want indexed.

If I understand you correctly by including the robots.txt on the pages I don't want found the internal search will ignore those pages when it someone types in a search for something if it comes across that?

3:36 pm on Jun 11, 2008 (gmt 0)

10+ Year Member



Google site search uses the Google index to find pages that it will use for your search results. If you block those pages from GoogleBot via robots.txt, they will not be in the Google index. So they will not show up in your site search results AND they will not show up when someone searches Google either.

I use Freefind for my site search (I can rebuild the site index any time I want), but I don't know if you can tell it not to add certain pages to the index.

9:43 am on Jun 13, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have used Swish-e for years. It's free, fast and no db required. Not sure about the exclusion options, but the site should tell you what you need to know. They also have a mailing list.

One install on a server lets you use it on any number of sites by adding a config file and a PERL script for each. Runs off a CRON. Try swish-e (dot) org

Another that is supposed to be even faster us Swish++, but I've never tried it. It's on Sourceforge.

7:13 pm on Jun 13, 2008 (gmt 0)

5+ Year Member



Is your site running off of a CMS (content management system) or is it just plain HTML pages? If your content grows large enough - I would recommend looking into developing your website on a CMS. Most all CMS systems have good search features built in. Then you don't have to rely on a service like Google.
9:57 pm on Jun 13, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thank you for the great tips. My site is currently just HTML pages, but I am looking at using a CMS for another site I am building for an unrelated business I am starting, which will have a lot more content and the need to be able to search it will be even more pronounced.
1:14 pm on Jun 17, 2008 (gmt 0)

5+ Year Member



If i have understood you correctly, i use a php mysql prog called sphider.
It spiders and indexes all your pages content and stores in the DB.. You can then query DB with search strings and returns list of results (think this is done using MATCH AGAINST). I have found it really useful.
Ally
1:59 pm on Jun 20, 2008 (gmt 0)

5+ Year Member



You can use Google Site Search, but why not include the Adsense one ? It's not a shame to make a few extra bucks from searches ;)
10:14 pm on Jun 20, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



but why not include the Adsense one ?

I know what Ad Sense is, but is the Ad Sense search tool? How does it work?

6:16 am on Jun 21, 2008 (gmt 0)

WebmasterWorld Administrator bill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The commercial version of Google site search costs about $100/year. That lets you show your chosen SERPs without AdSense. The free version of this tool requires that you show AdSense ads in the SERPs. I don't think that will be making you money.
3:47 pm on Jun 21, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We use the Adsense search box. Works quite well, though you do not get much control beyond robots.txt. Plus the extra income is nice.

At my last "real job" I installed a free search engine from Yahoo/IBM. Omni search or something like that. It was software based and worked quite well. And it was free.

1:48 am on Jun 24, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thank you for the suggestions. I will look into them, but the more I think about it the more I am thinking for this new project I will need a good CMS system which will have a built in search function.
2:00 pm on Jun 24, 2008 (gmt 0)

10+ Year Member



Sounds like a CMS is your ideal solution.

In the meantime, the suggestions about Goog site search are good. You should also take a peak at the open source PHP script called 'Sphider'.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month