Welcome to WebmasterWorld Guest from 54.226.130.194

Forum Moderators: not2easy & rumbas

Message Too Old, No Replies

Twitter Quietly Transitions from MySQL to a New Search Back End

     

tedster

2:37 am on Oct 7, 2010 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



  • Twitter's real-time search engine was, until very recently, based on the technology that Summize originally developed.

  • [Now we have] a new, modern search architecture based on a highly efficient inverted index instead of a relational database.

  • With over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day (!) we already put a very high load on our machines.

  • We estimate that we're only using about 5% of the available backend resources, which means we have a lot of headroom. Our new indexer could also index roughly 50 times more Tweets per second than we currently get!

    [engineering.twitter.com...]
  • phranque

    10:16 am on Oct 7, 2010 (gmt 0)

    WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



    it's based on lucene which is an open source apache project based on java.
    and they are contributing back to the trunk and the new real time branch.
    that's pretty cool.

    wheel

    11:29 am on Oct 7, 2010 (gmt 0)

    WebmasterWorld Senior Member wheel is a WebmasterWorld Top Contributor of All Time 10+ Year Member



    That is pretty cool. I've used lucene before, the fact that they're contributing to it is a good thing.

    Sgt_Kickaxe

    3:12 pm on Oct 7, 2010 (gmt 0)

    WebmasterWorld Senior Member sgt_kickaxe is a WebmasterWorld Top Contributor of All Time 5+ Year Member



    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).

    J_RaD

    5:12 pm on Oct 7, 2010 (gmt 0)

    WebmasterWorld Senior Member 5+ Year Member




    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).


    has it seen a drop in usage yet?

    weeks

    7:16 pm on Oct 7, 2010 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member



    A+ for technology, D- for providing something useful to humanity


    So, now we have a better filter to find those golden little nuggets among the tweets. Right?

    "How can I find some insights into xyz."
    "Search Twitter!"
    "Righto! Great idea!"

    I'm certain I'm wrong about this, but I cannot see this being useful to many folks.

    paulguy

    7:18 pm on Oct 7, 2010 (gmt 0)

    5+ Year Member



    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).


    I was of the same opinion for a while on Twitter. These days, I'm careful who I follow. Basically, I only follow people who know a lot more than I do about things I'm interested in. I actually get a lot out of Twitter that way.

    tedster

    2:14 am on Oct 8, 2010 (gmt 0)

    WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



    For market insights and new content ideas, try regular monitoring of some Twitter searches about your market's keywords, for example "anyone know" [keyword] - returns common questions that your site can answer.

    I just hope some of that new technical power to expand the time that data is available for search results.