homepage Welcome to WebmasterWorld Guest from 54.145.183.169
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt Validator bug?
Christian Storm from Turnitin Robot maintainers says my robots.txt is wrong
scottspence

10+ Year Member



 
Msg#: 100 posted 9:43 am on Dec 16, 2002 (gmt 0)

Hi,

Basically this is the problem, my robots.txt says this:

User-agent: *
Disallow: /*/pass/
Disallow: /noodle/
Disallow: bad.html

Which according to the protocol, as far as I can tell, is wrong. But it is approved by the validator. Google obeys it but the Turnitin Robot (and possibly others) do not.

I have made changes like this:

User-agent: *
Disallow: /noodle/
Disallow: /bad.html

i.e. the main issues seemed to be the wild card and the absence of the full path.

Any suggestions? - is there another more recent protocol that I am missing or is this a bug?

Cheers

Scott

PS I do hope this is the correct place to post this message!

 

martinibuster

WebmasterWorld Administrator martinibuster us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 100 posted 10:02 am on Dec 16, 2002 (gmt 0)

The validator has been known to be wrong [webmasterworld.com] in the past.

It's ok to be a little skeptical.

I don't use a robots.txt. I'm some people have valid reasons for using one for banning bad bots.

But if you're not banning bad bots, and simply telling bots to crawl you, I'd rather keep confusion at bay and not put one up.

That's just my way of doing things.

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 100 posted 11:15 am on Dec 17, 2002 (gmt 0)

>Which according to the protocol

I debated about that one for quite awhile. Is not necc wrong. As you stated, it is accepted by Google.

I went ahead and put it in as a warning instead of a full error.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved