startmebot

[keyplyr]

UA: Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
Protocol: HTTP/1.1
Robots.txt: No
Host: AWS
54.234.0.0 - 54.235.255.255
54.234.0.0/15

Our bot does not actually index any content, but only retrieves specific information from online sources. In doing this, our bot does not consider the contents of robots.txt

[lucy24]

our bot does not consider the contents of robots.txt

In that case, it's awfully considerate of them to (a) crawl from AWS and (b) use the element "Googlebot" in their UA while crawling from a non-Google IP. Saves supplemental blocking ;)

:: idly thinking how awkward it would be if we ever got a legitimate visitor from Montenegro* ::


* I had to look it up. Well, I knew it wasn't Maine.

[keyplyr]

Montenegro is the location of Casino Royal from the Daniel Craig residency of Bond films. Nice digs.