Welcome to WebmasterWorld Guest from 54.196.153.46

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt and allow and $

Advanced question

     
11:33 am on Sep 30, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 28, 2006
posts:63
votes: 0


Is it correct understood that the following robots.txt ONLY allows indexing of urls ending on =se and =no

User-agent: *
Allow: /*=se$
Allow: /*=no$
Disallow: /

12:21 pm on Sept 30, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 28, 2006
posts: 63
votes: 0


Well, just to make it simpler:

User-agent: *
Allow: /se/
Disallow: /

Everything in the /se/ folder allowed everything else not?

12:43 pm on Oct 1, 2009 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3120
votes: 111


simpler is generally better when it comes to robots.txt, i would go with your second version