| Robots.txt and allow and $
Advanced question |
djingo
msg:3998426 | 11:33 am on Sep 30, 2009 (gmt 0) |
Is it correct understood that the following robots.txt ONLY allows indexing of urls ending on =se and =no User-agent: *
Allow: /*=se$
Allow: /*=no$
Disallow: /
|
djingo
msg:3998459 | 12:21 pm on Sep 30, 2009 (gmt 0) |
Well, just to make it simpler: User-agent: *
Allow: /se/
Disallow: / Everything in the /se/ folder allowed everything else not?
|
goodroi
msg:3999213 | 12:43 pm on Oct 1, 2009 (gmt 0) |
simpler is generally better when it comes to robots.txt, i would go with your second version
|
|
|
