homepage Welcome to WebmasterWorld Guest from 54.167.174.90
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt question
excluding every folder expect one
JVB_Mktg

10+ Year Member



 
Msg#: 138 posted 7:22 pm on Mar 2, 2003 (gmt 0)

How can I have a bot ignore every directory in my site except one?

Do I have to disalow all of them 1 by 1 ie.

User-agent: bot
Disallow: /folder1/

User-agent: bot
Disallow: /folder2/

User-agent: bot
Disallow: /folder3/

User-agent: bot
Disallow: /folder4/ etc.. or is there an easier way.

I only need it to check www.mydomain.com/midi and exclude it from every other place.

TIA,

Javi

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 138 posted 7:33 pm on Mar 2, 2003 (gmt 0)

Javi,
Welcome to Webmaster World.

It is imperative that you keep in mind that robots.txt is a SUGGESTION to bots of compliance.
Many bad bots don't even read robots.txt.

With the above in mind. . .

User-agent: *
Disallow: /

denies all bots.
I seem to recall that Allow is not valid protocol in robots'txt.

So your alyernatives are either to deny all or list each "1 by 1"

JVB_Mktg

10+ Year Member



 
Msg#: 138 posted 7:46 pm on Mar 2, 2003 (gmt 0)

Thanks wilderness :)

weesnich

10+ Year Member



 
Msg#: 138 posted 7:47 pm on Mar 2, 2003 (gmt 0)

User-agent: *
Disallow: /folder1
Disallow: /folder2
Disallow: /folder3
Disallow: /folder4
[empty line here]

should work.

It is better to disallow "/folder", as some SE assume that they are allowed to try "/folder" if you only disallow "/folder/", especially if you have some internal or incoming links pointing to "/folder".

Weesnich

JVB_Mktg

10+ Year Member



 
Msg#: 138 posted 7:54 pm on Mar 2, 2003 (gmt 0)

Thanks weesnich, it looks a lot better now :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved