Welcome to WebmasterWorld Guest from 54.145.44.134

Forum Moderators: phranque

Message Too Old, No Replies

Robot.txt crawling erorr

     

dailypress

12:42 pm on May 2, 2014 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



hi,

can someone please explain what the robot.txt is and why is google webmaster giving me a crawling error stating that the robot.txt doesnt exist on example.com/robot.txt

thanks in advance,
DP

tangor

1:40 pm on May 2, 2014 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Even if it is blank, create a file named robots.txt and put it in your root directory.

For more info on what robots.txt is about, and its usage see [robotstxt.org...]

lucy24

6:12 pm on May 2, 2014 (gmt 0)

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



Not all "errors" are errors as we normally understand the term. Some things reported in wmt are informational: "Are you aware that you don't have a robots.txt file?"

afaik there is no difference between an empty robots.txt and one that doesn't exist (or that can't be found because it's at the wrong URL).

A robots.txt file with no content will definitely be smaller than your default 404 document, so it saves your server a few bytes. Other than that there's no reason to add one.

tangor

9:57 pm on May 2, 2014 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



The different is that even a blank robots.txt will shut WMT up, thus no more reporting errors. Agreed that there is no requirement for robots.txt on any website. Heck, I ran web sites for near 10 years before I even heard about robots.txt! :)

These days it does serve a semi-useful purpose, so learning about it is not a bad idea.