newbies - 12:54 am on Aug 9, 2013 (gmt 0)
I recently received several email from google webmaster tool notifying me "increase in server errors". Most of the errors I found in Webmaster tool are links to forum internal pages such as the following
I have been running the forum since the start of my site and have included in robots.txt the following when the forum was installed:
clearly, the urls google tries to access in vain are all excluded in robots.txt, why does google still want to crawl them?
BTW, I didn't have such problem a while ago, and no changes have been made to the forum and robots.txt ever since.