homepage Welcome to WebmasterWorld Guest from 54.197.110.151
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt validator bug
The robots.txt validator accepts malformed files (missing User-agent).
mortenf

10+ Year Member



 
Msg#: 87 posted 9:53 am on Nov 3, 2002 (gmt 0)

Hi there,

The robots.txt validator seems to have a small bug.

Trying to validate the file at [rdfig.xmlhack.com...] gives no errors, but the file only contains a single line:

Disallow:/search/

According to the specification, a record consists of at least on User-agent line and at least one Disallow line.

Shouldn't this file be flagged as in error?

Regards,
Morten Frederiksen

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 87 posted 10:10 pm on Nov 3, 2002 (gmt 0)

You are right - that should be invalid.

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 87 posted 10:17 pm on Nov 3, 2002 (gmt 0)

Should be set. Thanks! (I really enjoy finding new exceptions that the validator isn't handling just right).

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 87 posted 11:02 pm on Nov 3, 2002 (gmt 0)

Brett,

That reminds me... Did you see the post last month (or before) where a member reported that his problems with a French (IIRC) SE spider were caused by a missing "required" blank line after the final robots.txt record?

The quoted snippet from the spider operator claimed that the last line in the robots.txt file should be blank and have only a newline. I wish I could remember more...

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved