Welcome to WebmasterWorld Guest from 107.20.75.63

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt validator bug

The robots.txt validator accepts malformed files (missing User-agent).

     

mortenf

9:53 am on Nov 3, 2002 (gmt 0)

Inactive Member
Account Expired

 
 


Hi there,

The robots.txt validator seems to have a small bug.

Trying to validate the file at [rdfig.xmlhack.com...] gives no errors, but the file only contains a single line:

Disallow:/search/

According to the specification, a record consists of at least on User-agent line and at least one Disallow line.

Shouldn't this file be flagged as in error?

Regards,
Morten Frederiksen

10:10 pm on Nov 3, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


You are right - that should be invalid.
10:17 pm on Nov 3, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Should be set. Thanks! (I really enjoy finding new exceptions that the validator isn't handling just right).
11:02 pm on Nov 3, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Brett,

That reminds me... Did you see the post last month (or before) where a member reported that his problems with a French (IIRC) SE spider were caused by a missing "required" blank line after the final robots.txt record?

The quoted snippet from the spider operator claimed that the last line in the robots.txt file should be blank and have only a newline. I wish I could remember more...

Jim