Odd set of requests today for robots.txt from
47.209.81.47 (Optimum / Suddenlink)
73.221.86.216 (Comcast)
75.136.104.220 (Charter / Spectrum)
98.220.125.39 (Comcast)
108.31.181.219 (Verizon)
172.113.67.176 (Charter / Spectrum)
Where the UA's (respectively) are
Mozilla/5.0 (compatible; Lipperhey Site Explorer; ht tp:// www .lipperhey. com/)
Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; ht tp:// help.yahoo. co.jp/help/jp/search/indexing/indexing-15.html))
WhatsApp/2.16.13 A
Fyrebot/1.0
DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+ht tp:// help.goo. ne.jp/door/crawler.html)
Google-Adwords-Instant (+ht tp:// www. google. com/adsbot.html)
And the referers are:
htt ps:// www.bing. com/
htt ps:// www.yandex. com/
htt p:// www.baidu. com/
htt ps:// www.google. com/
Spur id's all IP's but one as a proxy. They did no crawling, just asked for robots.
