Best approach for searches within a website

Best solution to for a general search of both static and dynamic content

[ryan_b83]

I have been wondering for some time now, what is the best solution for a general search within a website. The options i thought of were:

-Google search within the site (assuming all your pages are indexed in Google)
-Spider your own site using something like PHP Dig. (good for spiering static text and dynamic content, but you need to re-index your site every time there is an update)
-Query the database on dynamic content (But won't find static content)

Andy ideas?

Thanks,
Ryan

[whoisgregg]

I used to use PHP dig, but for anything decent sized you end up wasting a lot of resources respidering the entire site every day when maybe all that's changed is a page or two.

I use google search and query the database for strings that match the pattern of being a SKU.

Whatever the database turns up I always put in as the top result(s) and blend the design to look like the rest of the SERPs. I then loop through the google results and remove the particular product page(s) if it would also show up there. (Otherwise the user sees duplicate results.)

I also "enhance" the results by looping through the google results and, when possible, adding a small thumbnail of the particular product that the user will find if they click on that URL.

The sites I do this for are particularly well spidered by Google, so it's unusual for a new page to take more than a few days to show up.

[ryan_b83]

Yea that sounds like a good idea. Do you use a specific Google API?

[whoisgregg]

Well, interestingly I just got an email from Google saying they are changing the whole site search program. I used to use Custom Search Business Edition which was (if I recall correctly) $99 per site, per year. The new program is called Google Site Search and has cost varying based on the number of web pages and the annual search query volume.

[henry0]

@whoisgregg

just checked it out
is the "more results..." part of the deal
if so how come do they ask for a fee when they also send your users to competitors!

[coopster]

I'm partial to sphinx search. It is lightning fast and you can tweak your index(es) with configuration files as well as your results with the provided APIs.

[ryan_b83]

@coopster

That sphinx search looks pretty cool, but what about for indexing static pages who's content is not stored in a database?

[coopster]

There is also an xml pipe interface, xmlpipe2.