Forum Moderators: open
If you're going to ask for robots.txt, you could at least use it BEFORE spidering (non-existent) dirs and files --
08:55:05 ... /iphone
08:55:06 ... /touch
08:55:07 ... /apple-touch-icon.png
08:55:08 ... /robots.txt
-- unless you're into crawling/causing 404s, that is. Hrrmph.
FWIW:
In Fall, 2005, "pat-tdc.opera.com" was a development server: "Opera Attack on Cryptome Was Opera Rogue Bot [cryptome.info]"
213.236.208. [14/Jun/2007] "GET /favicon.ico HTTP/1.0" 200 ? "-" "HTTP::Lite/2.1.6"
213.236.208. [05/Nov/2007] "GET / HTTP/1.1" 206 ? "-" "Opera/9.10 (Windows NT 5.1; U; en)"
213.236.208. [26/Jan/2008] "GET / HTTP/1.1" 404 ? "-" "W3C_Validator/1.575"
213.236.208. [06/Mar/2008] "GET / HTTP/1.1" 200 ? "-" "Opera/9.00 (X11; Linux i686; U; en)"
213.236.208. [06/Mar/2008] "GET /favicon.ico HTTP/1.1" 200 ? "http://www.example.com/" "Opera/9.00 (X11; Linux i686; U; en)"
213.236.208. [15/Apr/2008] "GET / HTTP/1.1" 200 ? "-" "Opera/9.00 (X11; Linux i686; U; en)"
213.236.208. [15/Apr/2008] "GET /favicon.ico HTTP/1.1" 200 ? "http://www.example.com/" "Opera/9.00 (X11; Linux i686; U; en)"
213.236.208. [06/Feb/2009] "GET /iphone HTTP/1.1" 404 ? "-" "Opera/9.10 (Windows NT 5.1; U; en)"
213.236.208. [06/Feb/2009] "GET /touch HTTP/1.1" 404 ? "-" "Opera/9.10 (Windows NT 5.1; U; en)"
213.236.208. [06/Feb/2009] "GET /apple-touch-icon.png HTTP/1.1" 404 ? "-" "Opera/9.10 (Windows NT 5.1; U; en)"
213.236.208. [06/Feb/2009] "GET /robots.txt HTTP/1.1" 200 ? "-" "Opera/9.10 (Windows NT 5.1; U; en)" 213.236.208. [14/Feb/2009:xx:xx:21] "GET /robots.txt HTTP/1.1" 200 ... "-" "Opera/9.10 (Windows NT 5.1; U; en)" User-agent: *
Disallow: /
213.236.208. [14/Feb/2009:xx:xx:21] "GET / HTTP/1.1" 403 ... "-" "Opera/9.10 (Windows NT 5.1; U; en)"
213.236.208. [14/Feb/2009:xx:xx:21] "GET / HTTP/1.1" 403 ... "-" "Opera/9.10 (Windows NT 5.1; U; en)" Range: bytes=0-999999