HubSpot Marketing Grader

[keyplyr]

UA: HubSpot Marketing Grader, HubSpot Marketing Grader
Protocol: HTTP/1.1
Robots.txt: No
Host: AWS-HUBSPOT
54.174.52.0 - 54.174.55.255
CIDR: 54.174.52.0/22

And yes the name is repeated
And yes the name is repeated

Related thread: [webmasterworld.com...]

[lucy24]

Robots.txt: No

Goodness, how persnickety you are ;) HubSpot does, too, ask for robots.txt ... many seconds after requesting (and getting 403'd on) the front page. They caught my attention because they went on to request all the supporting files that a human visitor would get with the 403 page. Also three separate 403d requests for the sitemap--which, unlike robots.txt, doesn't carry a blanket exemption, because why would it. In spite of the repeated sitemap requests, they never came back to request any of the pages linked from the 403 page.

I've seen it as
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler
and more recently as
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2764.0 Safari/537.36 HubSpot Website Grader ([e-mail])

I assume they're the same critter; I find them from 54.174.60.0/23 (or /22 or whatever the breakdown is).

[keyplyr]

"persnickety" I had to look that one up.

Just reporting them like I see them, but maybe I'm not adroitly looking for the robots.txt hit.

And yes it is /23 (typo)

[lucy24]

Since you met them from 52-55 and I found them from 60-61, they probably live in other places too. Or at least more of 54.174.
:: counting on fingers ::
At least 54.174.48.0/20
But then, a lot of AWS is distributed, isn't it.

:: wandering off to figure out why my IP chart has a white line running down the middle of it ::

[keyplyr]

Yes, cloud computing is distributed, depending on how the parent company offers the service to its customers, usually by server nodes.

Going forward, we should assume that agents coming from one cloud range could very well come from any other cloud range controlled by that parent host.