Twitter’s new Cards technology allows third-party developers to create richer, more compelling — and, above all, visually consistent — content inside of Twitter itself.
Therein lies Twitter’s goal: A rich, consistent Twitter experience for every user. When the hammer drops and Twitter changes its guidelines, those apps that can’t deliver this consistency will no longer be able to integrate with Twitter. The most likely candidates to go first, according to multiple sources, fall into two camps: Third-party-client apps which essentially reduplicate the Twitter stream — such as Tweetbot, Echofon and Osfoora — and news reader apps like Flipboard, which re-renders Twitter data to create a different visual experience of a tweet entirely.
The answer lies in the cards.
URL crawling
Twitter's crawler will respect robots.txt when scanning URLs. If a page with card markup is blocked, no card will be shown. If an image URL is blocked, no thumbnail or photo will be shown.
Twitter uses the User-Agent of Twitterbot/1.0, which can be used to create an exception in your robots.txt file. For example, here is a robots.txt which disallows crawling for all robots except Twitter's fetcher:
User-agent: Twitterbot
Disallow:
User-agent: *
Disallow: /
