homepage Welcome to WebmasterWorld Guest from 54.161.147.106
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Social Media / Twitter
Forum Library, Charter, Moderators: not2easy & rumbas

Twitter Forum

    
Twitter Quietly Transitions from MySQL to a New Search Back End
tedster




msg:4212738
 2:37 am on Oct 7, 2010 (gmt 0)

  • Twitter's real-time search engine was, until very recently, based on the technology that Summize originally developed.

  • [Now we have] a new, modern search architecture based on a highly efficient inverted index instead of a relational database.

  • With over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day (!) we already put a very high load on our machines.

  • We estimate that we're only using about 5% of the available backend resources, which means we have a lot of headroom. Our new indexer could also index roughly 50 times more Tweets per second than we currently get!

    [engineering.twitter.com...]

  •  

    phranque




    msg:4212933
     10:16 am on Oct 7, 2010 (gmt 0)

    it's based on lucene which is an open source apache project based on java.
    and they are contributing back to the trunk and the new real time branch.
    that's pretty cool.

    wheel




    msg:4212961
     11:29 am on Oct 7, 2010 (gmt 0)

    That is pretty cool. I've used lucene before, the fact that they're contributing to it is a good thing.

    Sgt_Kickaxe




    msg:4213120
     3:12 pm on Oct 7, 2010 (gmt 0)

    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).

    J_RaD




    msg:4213260
     5:12 pm on Oct 7, 2010 (gmt 0)


    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).


    has it seen a drop in usage yet?

    weeks




    msg:4213366
     7:16 pm on Oct 7, 2010 (gmt 0)

    A+ for technology, D- for providing something useful to humanity


    So, now we have a better filter to find those golden little nuggets among the tweets. Right?

    "How can I find some insights into xyz."
    "Search Twitter!"
    "Righto! Great idea!"

    I'm certain I'm wrong about this, but I cannot see this being useful to many folks.

    paulguy




    msg:4213369
     7:18 pm on Oct 7, 2010 (gmt 0)

    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).


    I was of the same opinion for a while on Twitter. These days, I'm careful who I follow. Basically, I only follow people who know a lot more than I do about things I'm interested in. I actually get a lot out of Twitter that way.

    tedster




    msg:4213551
     2:14 am on Oct 8, 2010 (gmt 0)

    For market insights and new content ideas, try regular monitoring of some Twitter searches about your market's keywords, for example "anyone know" [keyword] - returns common questions that your site can answer.

    I just hope some of that new technical power to expand the time that data is available for search results.

    Global Options:
     top home search open messages active posts  
     

    Home / Forums Index / Social Media / Twitter
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved