Alpha Build - New Search Engine!

[Brett_Tabke]

You are the first to see our new site search engine. This represents a couple months work and we are very proud of it:

[webmasterworld.com...]

Notes:
- we are not sold on the serp display yet, but we have a couple other features we want to add (thread mode vs message mode), so we are still working with templates.

Known issues:
- about 15minutes delayed on indexing new content

Anyway - we like it thus far and it is providing great results.

Thanks!

...Feedback, comments, wishes, and praise ;-) welcome and encouraged.

[trillianjedi]

Brett - great job. I'm impressed with the quality of results. That's the important thing. It's pretty fast too.

The cosmetics is the easy bit to fix up. I see lots of feedback in here already which is probably all you need to give it some tweaks. I'd throw it out to a wider audience before doing too much on that front though.

[mack]

I think if we look beyond the layout what we have is a great site search. Good results and very fast. The layout issues are potentially the simplest thing to alter, and the feedback on here will certainly help to refine that factor.

Mack.

[AnkitMaheshwari]

Its amazing....

[Ocean10000]

This is going to make my life a bit easier when doing research for User-Agents Histories and the like.

Good job.

[tangor]

I do like the advanced search very much. I'm also curious if a basic search at the top of every forum is also possible (with sub link to advanced to the side/below). This would make the search function more immediate and intimate with WW.

[idolw]

Great tool, thanks!
One thing I'd change is put the thread date closer to its title. Something like:
Alpha Build - New Search Engine! - Apr 11, 2010
Would be just perfect.

[inbound]

Well done Brett & team.

Getting such a search system to work well is not a trivial matter.

There are a few things that I would like to see change over time:

Like most, the presentation of the results needs attention but that's just some CSS really.

Addition of some further attributes such as total number of posts in the thread. These along with the presentation options should probably be user selectable from an options page - choose once and then always see the results as you wish. Other attributes could include, thread start date & last post date (however that probably involves keeping a separate index for the last post date and total post numbers as adding a post to a thread would then change attributes on many posts, so you would want to do a small second pass on those data items - which would also mean that some of the data would not be easy to incorporate into the ranking algo options - although being able to rank by post count in a thread may be useful occasionally).

Have a "member match?" option on the form so that you can choose between posts or member profiles that you are searching - or don't have the option and separate the 2 distinct result sets in some way (columns or stacked with internal anchor link to lower set)

It's a great start though.

[StoutFiles]

Any chance you'll share how the search used to work and how it works now? I'm always up for learning how to optimize queries.

[musicales]

Brett, echoing StoutFiles, you often said this was impossible given the nature of how the site is built (static files etc) - care to let us in on the technology you are using in the end to acheive this?

[oddsod]

Great job, I love it. Now I won'thave to go to Google to find a WW thread! :)

Would be good if you could format the output more like the SEs.

[Brett_Tabke]

StoutFIles - core db is modified Sphinx. There is another meg of our Perl code for a spider and ranking algo. Sphinx is awesome in that it doesn't require an sql db. We just feed it raw xml. However, there is a major ton of tweaking to get the level of relevance that we want.

> format the output more like the SEs.

Agreed - however, there are significant differences in the way we have built this SE vs the way big G/Y/Bing work. We are indexing down to the message level and yet, we need to return SERPS at the page/thread level. That leaves a wide array of interpretations on how the display should look. We also want to be able to toggle between two modes (thread/page mode vs message mode). Lastly, we are going to throw in a 3rd option and just allow people to open the message right on the SERP via ajax.

So, we are going to monkey with it a bit more this week.

[TheMadScientist]

That leaves a wide array of interpretations on how the display should look.

I was just going to post about the color and layout being all wrong...
It's not green and blue with lots of ads. What were you thinking? LOL

Actually, I think you all did a great job with it.
It had to be a huge (but hopefully fun) challenge. :)

[jecasc]

Wow - and it only took you how many years?

If webmasterworld can implement a site search - who nows what happens next. Maybe "Duke Nukem Forever" will be released after all.

[albo]

Superb!

[incrediBILL]

Works real nice.

Being able to search the supporters area is the best part!


I'd like to see date options to search for older documents and exclude newer ones such as

Posts > 6 Months
Posts > 1 Year
Posts > 2 Years

That way I could find older posts with sorting thru the current stuff, or as someone else suggested, allow sorting results by date.

[Sgt_Kickaxe]

I like it. The css for "member match" results could add an underline under those words to make it easier to see but the engine itself works great!

[Hawkgirl]

I really dig the member match - when I got my results and saw the member match, I thought, "Oh, cool, THAT's the guy to read."

[Readie]

There is one thing I dislike about this:

I can't search just by username (i.e. no other search criteria), and I feel that's an unecessary limitation. Other than that: awesome job guys :)

[tangor]

I'd like a member search tied to a date range. Oft times I'll remember who said something and about when, but not exactly what they said. :)

[Lorel]

I'm glad to see this Brett as it was annoying to have to go search elsewhere.

I'd like to see some color separations, maybe for title, like on the other pages.

[JohnRoy]

Wow, something we didn't dream will be available is here! Great job.

[Status_203]

Very similar spacing between "paras" of a single result and between one result and another (for me anyway (at work - Chrome 3, XP)). I'd prefer that the different parts of a result were closed up a bit to make this clearer.

Put together that which belongs together.

[BradleyT]

Searching for "Search Engine" gives pretty bad results.

[TheMadScientist]

Thanks a bunch for NOT eliminating 'stop words' from the searches... I just tried a search for information about a specific tld none of the other search engines I've tried with will display well in their results, but it was returned and highlighted here. YES!

None of the other search engines I've tried the search on even do very well with 'tld ".in" domain' as the search because they totally discount the '.' and the 'in' from the query, even if it's in quotes, so it's been difficult to find info about some tlds before.

Anyway, good thinkin' and THANKS!

[Jonesy]

"Membernames are case insensitive"

Doing a [check] on "Jonesy" == Member name not found

Doing a [check] on "jonesy" == Member name exists

Jonesy

[lavazza]

...Feedback, comments, wishes, and praise ;-) welcome and encouraged.

One wish:
Pass the keyword string (e.g. "Blue Widgets") into the resultant URI of the SERP

Doing so would make the new site search engine a useful tool when replying to posts that ask questions that have been comprehensively answered before

At the moment, a search for "Blue Widgets" generates a page with a (meaningless) URI of

http://www.webmasterworld.com/search.cgi

CF Google:

http://www.google.com/search?&q=site:www.webmasterworld.com+%22blue+widgets%22

[penders]

bill: I don't know if it's just me, but the images in the search form take forever to load. It's been like this since I was privy to the earlier release. Today it took 20 seconds for them all to load from my office. On my home fiber connection it takes close to 10 seconds.

About 3 seconds for me - but the images are certainly the slow bit. Do we need images here at all?

In the SERPs, sometimes there is no text snippet for some results and often the phrase I'm searching for does not appear in that text snippet (because it appears much futher down the post). I think I would prefer to see the text surrounding the search phrase in the SERPs.

>> Time range...
May be (to fit in with your current method), have an "excluding" drop down next to the current "Time Range" drop down. Both drop downs are essentially the same. So, you could pick "Time Range 'Last 5 Years' excluding the 'Last 2 Years'" ?

Great stuff - well done!

[yifancrusher]

Convenient,I like it very much.

[Propools]

[webmasterworld.com...] image is not showing up.

[Propools]

I did a check for under "User Name" for my user name.
I searched for my name with a capital "P" like it shows up on the screen display to the left.
<<<<<<<<<<<<<<<<<<<<<<<<<<<< To the left and up.

Well, it came back that there's no user by that name. I then tried the lower case "p" and I found myself again.
Whoo, I had an identity crisis there for a second.