JAB_Creations - 4:28 am on Sep 5, 2011 (gmt 0)

Do your robots.txt files utilize a database for any reason? If so do you record MySQL errors?

I started recording JavaScript, PHP and MySQL errors to logs months ago and I've noticed (and confirmed from talking with my current host) that they reset MySQL every so often. The error message generated is, "MySQL server has gone away". This may occur several times during the day, smack dab in the middle of the day in fact and the database may be unavailable for several seconds, long enough to cause someone somewhere some sort of trouble. I highly doubt the host I use is the only one who does this.

If your robots.txt file uses a service that temporarily becomes unavailable do you know how your site will react? In example by using the base element my entire site works seamlessly both locally and live with the exact same code; I have tested my site disabling the database and then loading pages (since my site is now database driven). My site actually is aware and able to fall back in to a "safe mode" of sorts if the database becomes unavailable for that request.

If that is the situation that's happening then I would recommend serving an HTTP 503 header presuming Google will wait a few moments/minutes before trying again instead of simply presuming what you're seeing.

- John