Welcome to WebmasterWorld Guest from 54.227.125.200

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt validator bug

The robots.txt validator accepts malformed files (missing User-agent).

     

mortenf

9:53 am on Nov 3, 2002 (gmt 0)



Hi there,

The robots.txt validator seems to have a small bug.

Trying to validate the file at [rdfig.xmlhack.com...] gives no errors, but the file only contains a single line:

Disallow:/search/

According to the specification, a record consists of at least on User-agent line and at least one Disallow line.

Shouldn't this file be flagged as in error?

Regards,
Morten Frederiksen

Brett_Tabke

10:10 pm on Nov 3, 2002 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



You are right - that should be invalid.

Brett_Tabke

10:17 pm on Nov 3, 2002 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Should be set. Thanks! (I really enjoy finding new exceptions that the validator isn't handling just right).

jdMorgan

11:02 pm on Nov 3, 2002 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Brett,

That reminds me... Did you see the post last month (or before) where a member reported that his problems with a French (IIRC) SE spider were caused by a missing "required" blank line after the final robots.txt record?

The quoted snippet from the spider operator claimed that the last line in the robots.txt file should be blank and have only a newline. I wish I could remember more...

Jim

 

Featured Threads

Hot Threads This Week

Hot Threads This Month