Forum Moderators: phranque
I want to find a solution capable of handling 1,000,000+ searches. Any and all help is greatly appreciated. thanks.
It's often the case that a simple page based search will not do (although there are great free and paid solutions out there if you just need a page based search you will have plenty of choice) - do you need to add other options to the search that makes it easier for people to narrow down the result set (think of how useless a Realty site would be without the options they have for location, price range, house type...).
Once you have decided what features the search should have (which is often good to do before starting coding as the database structure can massively effect the speed of retrieving information depending on your needs) you will need to look at what data that touches on and how resource intensive the searches are. If you can profile the likely mixture of search types you're on your way to getting a full picture of how your system is likely to perform (although you'll never get is spot on, people always do things you don't expect them to).
If you know the amount of work the system needs to do you can then evaluate different systems for handling parts of the search. Some search software delivers better results than others, but that may be at the expense of performance. Fortunately, 1 million searches a week isn't too many to cope with (but it's not a trivial amount) - if you take 'Standard' traffic patterns (yours may be different) you'll find that you'll see around 14,000 searches and hour as the maximum on a Monday around lunchtime - that's an average of 4 a second but you need to have the capability to serve more than that as the amount per second will vary quite a bit and you can have cascading effects of delays when you get too many queries hitting a system at the same time (you may want to quadruple that number to 16 per second to be more confident it should not fall over).
Are you sure about the 1 million a week or is that an estimate? The reason I ask is that building a good site search to 16 queries a second may be expensive (depending on the data being searched).
I know this may have brought up more questions than answers, but that's the nature of site search - it's worth putting a fair bit of planning time into but that often leads to more difficult implementation requirements (the key here is to have a search that people like using, so they come back).
Best of luck with the project.
Using G's Adsense search on our site pays out A very good CPM. I know rates vary, but even at $10 CPM that could mean $40k/month rev share in your pocket.
And a pretty good free search tool on top of that. ;-)
