homepage Welcome to WebmasterWorld Guest from 54.196.62.23
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Webmaster General
Forum Library, Charter, Moderators: phranque

Webmaster General Forum

    
Robot.txt crawling erorr
dailypress

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4667950 posted 12:42 pm on May 2, 2014 (gmt 0)

hi,

can someone please explain what the robot.txt is and why is google webmaster giving me a crawling error stating that the robot.txt doesnt exist on example.com/robot.txt

thanks in advance,
DP

 

tangor

WebmasterWorld Senior Member tangor us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4667950 posted 1:40 pm on May 2, 2014 (gmt 0)

Even if it is blank, create a file named robots.txt and put it in your root directory.

For more info on what robots.txt is about, and its usage see [robotstxt.org...]

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4667950 posted 6:12 pm on May 2, 2014 (gmt 0)

Not all "errors" are errors as we normally understand the term. Some things reported in wmt are informational: "Are you aware that you don't have a robots.txt file?"

afaik there is no difference between an empty robots.txt and one that doesn't exist (or that can't be found because it's at the wrong URL).

A robots.txt file with no content will definitely be smaller than your default 404 document, so it saves your server a few bytes. Other than that there's no reason to add one.

tangor

WebmasterWorld Senior Member tangor us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4667950 posted 9:57 pm on May 2, 2014 (gmt 0)

The different is that even a blank robots.txt will shut WMT up, thus no more reporting errors. Agreed that there is no requirement for robots.txt on any website. Heck, I ran web sites for near 10 years before I even heard about robots.txt! :)

These days it does serve a semi-useful purpose, so learning about it is not a bad idea.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Webmaster General
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved