dstiles - 9:39 pm on Feb 11, 2013 (gmt 0)

Wilderness - 500 says the server has a fault. Since this may be a temporary off-line thing (eg reboot of MS IIS server after update) the bot should re-visit within a reasonable time to ensure the 500 is not permanent, which could occur if the server were taken off-line permanently (you've moved your site elsewhere but forgotten to tell DNS).

Often the 500 is merely "I'm a bit busy at the moment" - too much traffic to handle.