Welcome to WebmasterWorld Guest from 54.198.100.0

Forum Moderators: not2easy & rumbas

Message Too Old, No Replies

Twitter Quietly Transitions from MySQL to a New Search Back End

     
2:37 am on Oct 7, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


  • Twitter's real-time search engine was, until very recently, based on the technology that Summize originally developed.

  • [Now we have] a new, modern search architecture based on a highly efficient inverted index instead of a relational database.

  • With over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day (!) we already put a very high load on our machines.

  • We estimate that we're only using about 5% of the available backend resources, which means we have a lot of headroom. Our new indexer could also index roughly 50 times more Tweets per second than we currently get!

    [engineering.twitter.com...]
  • 10:16 am on Oct 7, 2010 (gmt 0)

    Administrator

    WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

    joined:Aug 10, 2004
    posts:10553
    votes: 13


    it's based on lucene which is an open source apache project based on java.
    and they are contributing back to the trunk and the new real time branch.
    that's pretty cool.
    11:29 am on Oct 7, 2010 (gmt 0)

    Senior Member

    WebmasterWorld Senior Member wheel is a WebmasterWorld Top Contributor of All Time 10+ Year Member

    joined:Feb 11, 2003
    posts:5069
    votes: 12


    That is pretty cool. I've used lucene before, the fact that they're contributing to it is a good thing.
    3:12 pm on Oct 7, 2010 (gmt 0)

    Senior Member

    WebmasterWorld Senior Member sgt_kickaxe is a WebmasterWorld Top Contributor of All Time 5+ Year Member

    joined:Apr 14, 2010
    posts:3169
    votes: 0


    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).
    5:12 pm on Oct 7, 2010 (gmt 0)

    Senior Member

    WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

    joined:Jan 30, 2006
    posts:1655
    votes: 7



    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).


    has it seen a drop in usage yet?
    7:16 pm on Oct 7, 2010 (gmt 0)

    Senior Member

    WebmasterWorld Senior Member 10+ Year Member

    joined:Jan 22, 2005
    posts:1163
    votes: 6


    A+ for technology, D- for providing something useful to humanity


    So, now we have a better filter to find those golden little nuggets among the tweets. Right?

    "How can I find some insights into xyz."
    "Search Twitter!"
    "Righto! Great idea!"

    I'm certain I'm wrong about this, but I cannot see this being useful to many folks.
    7:18 pm on Oct 7, 2010 (gmt 0)

    New User

    5+ Year Member

    joined:June 11, 2010
    posts: 18
    votes: 0


    A+ for technology, D- for providing something useful to humanity (unless you like mostly regurgitated spam everywhere you look).


    I was of the same opinion for a while on Twitter. These days, I'm careful who I follow. Basically, I only follow people who know a lot more than I do about things I'm interested in. I actually get a lot out of Twitter that way.
    2:14 am on Oct 8, 2010 (gmt 0)

    Senior Member

    WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

    joined:May 26, 2000
    posts:37301
    votes: 0


    For market insights and new content ideas, try regular monitoring of some Twitter searches about your market's keywords, for example "anyone know" [keyword] - returns common questions that your site can answer.

    I just hope some of that new technical power to expand the time that data is available for search results.