homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Webmaster General
Forum Library, Charter, Moderators: phranque & physics

Webmaster General Forum

Robot.txt crawling erorr

 12:42 pm on May 2, 2014 (gmt 0)


can someone please explain what the robot.txt is and why is google webmaster giving me a crawling error stating that the robot.txt doesnt exist on example.com/robot.txt

thanks in advance,



 1:40 pm on May 2, 2014 (gmt 0)

Even if it is blank, create a file named robots.txt and put it in your root directory.

For more info on what robots.txt is about, and its usage see [robotstxt.org...]


 6:12 pm on May 2, 2014 (gmt 0)

Not all "errors" are errors as we normally understand the term. Some things reported in wmt are informational: "Are you aware that you don't have a robots.txt file?"

afaik there is no difference between an empty robots.txt and one that doesn't exist (or that can't be found because it's at the wrong URL).

A robots.txt file with no content will definitely be smaller than your default 404 document, so it saves your server a few bytes. Other than that there's no reason to add one.


 9:57 pm on May 2, 2014 (gmt 0)

The different is that even a blank robots.txt will shut WMT up, thus no more reporting errors. Agreed that there is no requirement for robots.txt on any website. Heck, I ran web sites for near 10 years before I even heard about robots.txt! :)

These days it does serve a semi-useful purpose, so learning about it is not a bad idea.

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Webmaster General
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved