Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Pattern Matching in Robots.txt !

Can someone please clarify a few points for me?

4:07 am on Jun 22, 2010 (gmt 0)

New User

5+ Year Member

joined:June 22, 2010
votes: 0

I created the following Robots.txt file and want to block access to URLs ending in a certain pattern.

The URLs I want to block look like this:

http: //www.XYZ.com/-white/car.html
http: //www.XYZ.com/-blue/car.html

I dont want to block any URL that looks like this:

http: //www.XYZ.com/white-car.html
http: //www.XYZ.com/blue-car.html

so I have my robots.txt file as follows:

User-agent: *
Disallow: /*/car.html$

So this should only block URLs ending with " /car.html "
I want to make sure it only blocks URLs that have a " / " right before the "car.html" and not ones ending like" -car.html"

Can you guys let me know if I have this concept correct.
I would appreciate all feed back.

1:28 am on June 23, 2010 (gmt 0)


WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
votes: 126

welcome to WebmasterWorld [webmasterworld.com], James!

there is a robots.txt test function available in GWT, described at the bottom of this page:
6:22 am on Aug 16, 2010 (gmt 0)

New User

5+ Year Member

joined:Aug 4, 2010
votes: 0

Hey, James -

If you want your pages blocked completely, you may want to go with Noindex...Disallowed pages can still be accessed by spiders, albeit in a very limited capacity.