homepage Welcome to WebmasterWorld Guest from 107.22.45.61
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt
How to make a valid positive list?
matthias




msg:1525761
 12:00 pm on Jul 7, 2002 (gmt 0)

It is possible to write a robots.txt file that validates and only allow some spiders?

I would write somthing like:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

but it does not validate.

 

jdMorgan




msg:1525762
 12:48 am on Jul 8, 2002 (gmt 0)

matthias,

That's weird... It should work fine. What validator are you using?

How about:

User-agent: Googlebot
Disallow: /some_directory_that_doesn't_exist

User-agent: *
Disallow: /

That might be a work-around.

Two points, just in case: You must have a blank line between records (After a
Disallow line and before the next User-agent line). And you need to write your
robots.txt file using an editor that will not put carriage-return/linefeed pairs
at the end of each line. robots.txt is a Unix file, and it only wants to see a
linefeed. I use Microsoft Word in ASCII mode with "Linefeed Only" selected
when I have to work on a robots.txt file on a Windows sytem. These options
are available in Word's "Save As" dialog box. The Notepad editor will put
CR/LF pairs in the file, and it won't validate.

Jim

matthias




msg:1525763
 12:07 am on Jul 9, 2002 (gmt 0)

I use [searchengineworld.com ]

I found the error. I had somthing like this on top of the file:

User-agent: *
Disallow: /trap/

Now I think I have to delete that one and instead replace every

Disallow:

with

Disallow: /trap/

not pretty if you ask me, but...

jdMorgan




msg:1525764
 12:38 am on Jul 9, 2002 (gmt 0)

matthias,

Glad you found it!

Jim

matthias




msg:1525765
 6:20 am on Jul 9, 2002 (gmt 0)

I'm glad to :-)

and thanks btw

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved