can someone please explain what the robot.txt is and why is google webmaster giving me a crawling error stating that the robot.txt doesnt exist on example.com/robot.txt
thanks in advance, DP
tangor
1:40 pm on May 2, 2014 (gmt 0)
Even if it is blank, create a file named robots.txt and put it in your root directory.
For more info on what robots.txt is about, and its usage see [robotstxt.org...]
lucy24
6:12 pm on May 2, 2014 (gmt 0)
Not all "errors" are errors as we normally understand the term. Some things reported in wmt are informational: "Are you aware that you don't have a robots.txt file?"
afaik there is no difference between an empty robots.txt and one that doesn't exist (or that can't be found because it's at the wrong URL).
A robots.txt file with no content will definitely be smaller than your default 404 document, so it saves your server a few bytes. Other than that there's no reason to add one.
tangor
9:57 pm on May 2, 2014 (gmt 0)
The different is that even a blank robots.txt will shut WMT up, thus no more reporting errors. Agreed that there is no requirement for robots.txt on any website. Heck, I ran web sites for near 10 years before I even heard about robots.txt! :)
These days it does serve a semi-useful purpose, so learning about it is not a bad idea.
