Has anyone else noticed random robots requesting robots.txt and giving “https://www.google.com/” as referer?
It started pretty suddenly in late September and has been ongoing since them. Random robots from assorted AWS neighborhoods, sporting humanoid but wildly antiquated UAs. My favorite--possibly theirs too, because it shows up a lot--is
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:188.8.131.52) Gecko/20080219 Firefox/184.108.40.206 Navigator/220.127.116.11
Logged headers tell me they're especially concerned with getting a fresh
Just robots.txt. Never anything else. Do they not realize that sending a bogus referer with a robots.txt request is more
likely to attract attention? Won't get them blocked, because I've got an ironclad policy of letting everyone
see robots.txt, no exceptions, but honestly. This is silly.