TheMadScientist - 5:42 pm on Dec 12, 2012 (gmt 0) [edited by: tedster at 8:14 pm (utc) on Dec 12, 2012]
HUH? The robot isn't what can't find something, it's the server that can't find the explicit information relating to, or specific resource associated with, a particular URL, so it serves the robot a default page or information with a status code indicating it's status for the request made by the robot.
Even if the page is completely blank in a browser, because they don't show you the status code and other server headers, there is still information served by a properly functioning server for Any request made.
IOW: A bot Always finds something for Every request made to a properly functioning server.
[edited by: tedster at 8:14 pm (utc) on Dec 12, 2012]