Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Robot.txt crawling erorr dailypress msg:4667952 12:42 pm on May 2, 2014 (gmt 0) hi, can someone please explain what the robot.txt is and why is google webmaster giving me a crawling error stating that the robot.txt doesnt exist on example.com/robot.txt thanks in advance, DP
tangor msg:4667978 1:40 pm on May 2, 2014 (gmt 0)
Even if it is blank, create a file named robots.txt and put it in your root directory. For more info on what robots.txt is about, and its usage see [ ...] robotstxt.org lucy24 msg:4668064 6:12 pm on May 2, 2014 (gmt 0)
Not all "errors" are errors as we normally understand the term. Some things reported in wmt are informational: "Are you aware that you don't have a robots.txt file?" afaik there is no difference between an empty robots.txt and one that doesn't exist (or that can't be found because it's at the wrong URL). A robots.txt file with no content will definitely be smaller than your default 404 document, so it saves your server a few bytes. Other than that there's no reason to add one. tangor msg:4668097 9:57 pm on May 2, 2014 (gmt 0)
The different is that even a blank robots.txt will shut WMT up, thus no more reporting errors. Agreed that there is no requirement for robots.txt on any website. Heck, I ran web sites for near 10 years before I even heard about robots.txt! :) These days it does serve a semi-useful purpose, so learning about it is not a bad idea.