homepage Welcome to WebmasterWorld Guest from 54.166.128.254
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Social Media / Twitter
Forum Library, Charter, Moderators: not2easy & rumbas

Twitter Forum

    
Twitter Quietly Transitions from MySQL to a New Search Back End
tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4212736 posted 2:37 am on Oct 7, 2010 (gmt 0)

  • Twitter's real-time search engine was, until very recently, based on the technology that Summize originally developed.

  • [Now we have] a new, modern search architecture based on a highly efficient inverted index instead of a relational database.

  • With over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day (!) we already put a very high load on our machines.

  • We estimate that we're only using about 5% of the available backend resources, which means we have a lot of headroom. Our new indexer could also index roughly 50 times more Tweets per second than we currently get!

    [engineering.twitter.com...]

  •  

    phranque

    WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



     
    Msg#: 4212736 posted 10:16 am on Oct 7, 2010 (gmt 0)

    it's based on lucene which is an open source apache project based on java.
    and they are contributing back to the trunk and the new real time branch.
    that's pretty cool.

    wheel

    WebmasterWorld Senior Member wheel us a WebmasterWorld Top Contributor of All Time 10+ Year Member



     
    Msg#: 4212736 posted 11:29 am on Oct 7, 2010 (gmt 0)

    That is pretty cool. I've used lucene before, the fact that they're contributing to it is a good thing.

    Sgt_Kickaxe

    WebmasterWorld Senior Member sgt_kickaxe us a WebmasterWorld Top Contributor of All Time



     
    Msg#: 4212736 posted 3:12 pm on Oct 7, 2010 (gmt 0)

    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).

    J_RaD

    WebmasterWorld Senior Member 5+ Year Member



     
    Msg#: 4212736 posted 5:12 pm on Oct 7, 2010 (gmt 0)


    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).


    has it seen a drop in usage yet?

    weeks

    WebmasterWorld Senior Member 5+ Year Member



     
    Msg#: 4212736 posted 7:16 pm on Oct 7, 2010 (gmt 0)

    A+ for technology, D- for providing something useful to humanity


    So, now we have a better filter to find those golden little nuggets among the tweets. Right?

    "How can I find some insights into xyz."
    "Search Twitter!"
    "Righto! Great idea!"

    I'm certain I'm wrong about this, but I cannot see this being useful to many folks.

    paulguy



     
    Msg#: 4212736 posted 7:18 pm on Oct 7, 2010 (gmt 0)

    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).


    I was of the same opinion for a while on Twitter. These days, I'm careful who I follow. Basically, I only follow people who know a lot more than I do about things I'm interested in. I actually get a lot out of Twitter that way.

    tedster

    WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



     
    Msg#: 4212736 posted 2:14 am on Oct 8, 2010 (gmt 0)

    For market insights and new content ideas, try regular monitoring of some Twitter searches about your market's keywords, for example "anyone know" [keyword] - returns common questions that your site can answer.

    I just hope some of that new technical power to expand the time that data is available for search results.

    Global Options:
     top home search open messages active posts  
     

    Home / Forums Index / Social Media / Twitter
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved