Had a visitor come on the site today calling itself the BOTW Spider, but there was no referer and the IP belongs to Jurgle LLC in New York, according to ARIN. Reverse dns says: spider.botw.org. The IP is 209.73.194.nn.
Is this a BOTW spider or someone else? Did not go for robots.txt It was spidering a particular html page. Only the page, no images.
Here is the hit from the logs:
209.73.194.nn - - [24/Aug/2011:07:00:35 -0400] "GET /html page HTTP/1.1" 200 5067 "-" "Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)"
From dawhois.com: BOTW.org, Organization: Jurgle LLC, ISP: Quality Technology Services.
Is it legit?
grandma_genie
