Web servers run separate logs that show web site errors. These logs can show things like access to robots.txt (if it doesn't exist), and cgi program failures. See Also: Server CGI robots.txt
See Also: Server CGI robots.txt