| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Pattern Matching in Robots.txt !|
Can someone please clarify a few points for me?
| 4:07 am on Jun 22, 2010 (gmt 0)|
I created the following Robots.txt file and want to block access to URLs ending in a certain pattern.
The URLs I want to block look like this:
I dont want to block any URL that looks like this:
so I have my robots.txt file as follows:
So this should only block URLs ending with " /car.html "
I want to make sure it only blocks URLs that have a " / " right before the "car.html" and not ones ending like" -car.html"
Can you guys let me know if I have this concept correct.
I would appreciate all feed back.
| 1:28 am on Jun 23, 2010 (gmt 0)|
welcome to WebmasterWorld [webmasterworld.com], James!
there is a robots.txt test function available in GWT, described at the bottom of this page:
| 6:22 am on Aug 16, 2010 (gmt 0)|
Hey, James -
If you want your pages blocked completely, you may want to go with Noindex...Disallowed pages can still be accessed by spiders, albeit in a very limited capacity.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved