Forum Moderators: open

Message Too Old, No Replies

bitlybot

AWS again

         

GaryK

5:39 pm on Jul 26, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



bitlybot
75.101.136.nnn
ec2-75-101-136-nnn.compute-1.amazonaws.com
-----
OrgName: Amazon.com, Inc.
OrgID: AMAZO-4
Address: Amazon Web Services, Elastic Compute Cloud, EC2
NetRange: 75.101.128.0 - 75.101.255.255

ROBOTS.TXT? No

Took the default root page for my browser project, the only site anything using AWS can access.

Bitlybot sounded so familiar to me at first. Then I realized I was thinking of the URL shortening service bit.ly. I wonder if there's any connection. I couldn't find anything connecting the two using Google though.

keyplyr

12:33 am on Jul 27, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Yes, it's the Twitter short URL utility, not really a web crawler so robots.txt may not be applicable. Many of these new Twitter apps use Amazon Web Service which is why I let up my EC2 IP range ban. Twitter generates a lot of traffic for me lately.

GaryK

2:43 am on Jul 27, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Can you explain to me why bit.ly needs a bot of any kind? Does it go get the URL someone is making a short link to?

keyplyr

7:24 am on Jul 27, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I believe it archives for its other services.