I'm a java developer with a fortune 50 retail company. Our business isn't generated from the web, but I thought of no better a place to pose this business problem than to those whose business is the web. I'm sure you guys can help me. One of our teams has developed a search engine that has much to be desired, I've been tasked with re-architecting it. I'll try to give a high level description of the design. I've got ideas as to the approach I'm going to take with this, but I'd like some fresh takes on the subject from you guys.
- First, relational data (over 1 million rows in some tables) is sent through the pipe to a localized database server.
- This data is queried for all fields that make up the search string for each business unit and a space delimited flat string is created for each.
- Simple words like 'a', 'the', and 'or' are stripped from each search string.
- Then, a row is added to a new table for each word in the business unit's search string (this was done to avoid the use of LIKE in our SQL, instead we can do a quicker ='keyword' on the table we just created)
- Finally, each time a user performs a search, now they only have one flattened table to query.
As you can see, this is an overly-complicated approach. Proof that it isn't designed well is that each keyword search takes 1-2 minutes. Let me know what you guys think.
Thanks,
JavaGuy
