m5hosting

faking UAs

[keyplyr]

UA: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36
UA: FacebookExternalHit/1.1
Protocol: HTTP/1.1
Robots.txt: No
Host: M5 Computer Security (m5hosting.com)
206.251.255.0 - 206.251.255.255
206.251.255.0/24
Parent: American Internet Services (americanis.net)
206.251.224.0 - 206.251.255.255
206.251.224.0/19

[lucy24]

UA: FacebookExternalHit/1.1

In CamelCase, like that? Well, that’s easily dealt with ;)

[keyplyr]

3 to 10 hits with the Mac UA, then once with the Facebook, then again & again. I had the range blocked already so I don't know if behavior would change otherwise.

I also filter anything Facebook by range.

[lucy24]

Incidentally ... It occurs to me that it might be useful to know what these various robots do--other than ask for, or not ask for, robots.txt. If it swings by, requests the front page and nothing else, that's one thing. If it homes in on some seemingly random interior page, that's quite another thing. And if it shows up out of nowhere and promptly does a full spidering--assuming it's able to get that far--that's a third and vastly horribler thing.

[keyplyr]

Well that's the discussion part. I usually just stick to documentation unless someone requests more info.

However... this one did a vastly horribler full crawl, but got all 403s. That says a lot in itself; programed to request all files regardless of server response.

[keyplyr]

Hitting one of the sites I manage about 3k to 5k daily, all faked UAs, same range.