Twitter’s new Cards technology allows third-party developers to create richer, more compelling — and, above all, visually consistent — content inside of Twitter itself.
Therein lies Twitter’s goal: A rich, consistent Twitter experience for every user. When the hammer drops and Twitter changes its guidelines, those apps that can’t deliver this consistency will no longer be able to integrate with Twitter. The most likely candidates to go first, according to multiple sources, fall into two camps: Third-party-client apps which essentially reduplicate the Twitter stream — such as Tweetbot, Echofon and Osfoora — and news reader apps like Flipboard, which re-renders Twitter data to create a different visual experience of a tweet entirely.
Msg#: 4481022 posted 6:24 am on Aug 2, 2012 (gmt 0)
Can't have reduplicated reduplication on the net now can we. They need to be careful not to complicate things AT ALL because many celebrities like it for its simplicity. Celebs like @Alyssa_Milano are into a different sort of card playing and arent likely going to want to spend time maintaining a more complicated system.
Msg#: 4481022 posted 10:22 pm on Aug 2, 2012 (gmt 0)
Note that they claim Twitterbot honors robots.txt!
Twitter's crawler will respect robots.txt when scanning URLs. If a page with card markup is blocked, no card will be shown. If an image URL is blocked, no thumbnail or photo will be shown.
Twitter uses the User-Agent of Twitterbot/1.0, which can be used to create an exception in your robots.txt file. For example, here is a robots.txt which disallows crawling for all robots except Twitter's fetcher: