homepage Welcome to WebmasterWorld Guest from 54.146.190.193
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Social Media / Twitter
Forum Library, Charter, Moderators: not2easy & rumbas

Twitter Forum

    
Twitter Search Uses Real-Time Human Computation
engine

WebmasterWorld Administrator engine us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month



 
Msg#: 4534722 posted 5:44 pm on Jan 9, 2013 (gmt 0)

Twitter's engineering department has explained how it uses humans to help improve its search. That's interesting on several points, and is probabably the opposite of Google's automated system.

First, we monitor for which search queries are currently popular.
Behind the scenes: we run a Storm topology that tracks statistics on search queries.
For example, the query [Big Bird] may suddenly see a spike in searches from the US.


As soon as we discover a new popular search query, we send it to our human evaluators, who are asked a variety of questions about the query.
Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to Amazon's Mechanical Turk service, and then polls Mechanical Turk for a response.
For example: as soon as we notice "Big Bird" spiking, we may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant Tweets and ads.


Finally, after a response from an evaluator is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our evaluators tell us that [Big Bird] is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.

Twitter Search Uses Real-Time Human Computation [engineering.twitter.com]

 

Sgt_Kickaxe

WebmasterWorld Senior Member sgt_kickaxe us a WebmasterWorld Top Contributor of All Time



 
Msg#: 4534722 posted 8:42 pm on Jan 9, 2013 (gmt 0)

I'm not sure it's the opposite of what Google does, I think they've already mapped most search terms as Twitter seems to be doing now, and Google has incorporated ad results and many other factors into their results. Not only that but Google is tracking individuals in order to optimize results on a user by user basis which is light years ahead of what twitter is describing.

Still, it's nice to see Twitter formulate what might also become an independent standalone search engine, building the keyword map is a major part of creating a viable search engine and they do have a unique source of data.

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4534722 posted 12:09 am on Jan 10, 2013 (gmt 0)

They use humans?

OH NO! IT'S SOYLENT SEARCH!

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4534722 posted 1:11 am on Jan 10, 2013 (gmt 0)

Not only that but Google is tracking individuals in order to optimize results on a user by user basis

Yah, but they're not using humans to do it. Unless they've discovered that outsourcing to {insert country name} is cheaper than doing it by computer. So all those weirdnesses we've been ascribing to Computer Glitch are really Non-Native-Speaker Misunderstanding.

Soylent Search eh. Personally I was impressed by the Mechanical Turk reference.

Q.: What percentage of Twitter users would respond to both phrases with "Huh?" or equivalent?

Inevitable corollary Q.: What percentage of WebmasterWorld readers ... et cetera, as above ;)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Social Media / Twitter
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved