Forum Moderators: open

Message Too Old, No Replies

TwitterBot

         

incrediBILL

7:44 am on Dec 11, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



It did ask for robots.txt (gasp) and has 3 IPs with rDNS best I can tell.

Twitterbot/1.0
199.59.149.164 -> spiderduck01.dmz1.twitter.com
199.59.149.165 -> spiderduck02.dmz1.twitter.com
199.59.149.166 -> spiderduck03.dmz1.twitter.com

But I've seen it crawl from all these too:

199.59.149.1 -> r-199-59-149-1.twttr.com
199.59.149.2 -> ditto, etc.
199.59.149.3
199.59.149.4
199.59.149.5
199.59.149.8
199.59.149.9

However, they have another variant that only does HEAD requests and I've seen on the following IPs:

Twitterbot/0.1
199.59.149.3 -> r-199-59-149-3.twttr.com
199.59.149.1 -> r-199-59-149-1.twttr.com
199.59.149.10 -> r-199-59-149-10.twttr.com

keyplyr

8:14 am on Dec 11, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If a link to your site is posted on Twitter, this and a dozen others will swarm in the matter of seconds. Reminds me a of what Piranha do to calf crossing a stream.

Pfui

1:25 pm on Dec 11, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



See also: "Twitter's Real Time URL Fetcher: SpiderDuck" [webmasterworld.com...]