Multiple User Agents after one page at the same time.

Twitterbot, Topsy, Tweetmemebot

[grandma genie]

Just thought I would share this with you all, since I've never seen anything like it. I was following an IP because it seemed to be static and ended up finding a whole series of bots hitting the same page, all at the same time. Many of them had already been blocked in htaccess, but this just gave me an excuse to add some more. I did not block the Googlebot, Slurp or Bing, but thought it was odd that they, too, were checking out the same page. I noticed the static IP (174.17.232.nnn) kept changing its user agent.

174.17.232.nnn "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"
128.242.241.nnn - HTTP/1.1" 200 - "-" "Twitterbot/0.1"
68.233.225.nn -"http://twitter.com/Obonic/status/4640044845367297" "Jaxified Bot (+http://www.jaxified.com/crawler/)"
66.249.65.nnn - HTTP/1.1" 200 5152 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
67.195.115.nn - HTTP/1.0" 200 5152 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]
65.52.2.nn - HTTP/1.1" 200 5152 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
216.24.142.nn - HTTP/1.0" 200 5152 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 OneRiot/1.0 (http://www.oneriot.com)"
184.73.68.nn - HTTP/1.1" 403 - "-" "PycURL/7.18.2"
76.118.162.nnn - HTTP/1.1" 200 5152 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"
74.112.128.nn - "GET /robots.txt HTTP/1.0" 200 3304 "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8"
89.151.116.nn - HTTP/1.1" 200 5152 "-" "Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot"
174.17.232.nnn - HTTP/1.1" 403 334 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101013 Ubuntu/9.04 (jaunty) Firefox/3.6.11"
204.236.254.nnn - - [16/Nov/2010:16:01:33 -0500] HTTP/1.1" 403 - "-" "PostRank/2.0 (postrank.com)"
38.113.234.nnn - HTTP/1.1" 403 334 "-" "Voyager/1.0"

[keyplyr]

Looks like a Twitter swarm.

A link to a file on your domain was posted on Twitter and has attracted all the usual suspects.

[grandma genie]

Ahhhh... Well that's a new one on me. Kinda like the sound of it, though. Twitter swarm. You learn something new every day. Thank you keyplyr.

[keyplyr]

RE: Twitter Swarm

I believe the citation goes to Pfui

[tangor]

Take note of the "suspects", chose those you keep and nuke the rest because this will only get worse before it becomes a nightmare. Pick and chose your "fave" bots. And my "faves" will be different from yours.