Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Pattern Matching in Robots.txt !


james22 - 4:07 am on Jun 22, 2010 (gmt 0)


I created the following Robots.txt file and want to block access to URLs ending in a certain pattern.

The URLs I want to block look like this:

http: //www.XYZ.com/-white/car.html
http: //www.XYZ.com/-blue/car.html


I dont want to block any URL that looks like this:

http: //www.XYZ.com/white-car.html
http: //www.XYZ.com/blue-car.html

so I have my robots.txt file as follows:

User-agent: *
Disallow: /*/car.html$

So this should only block URLs ending with " /car.html "
I want to make sure it only blocks URLs that have a " / " right before the "car.html" and not ones ending like" -car.html"

Can you guys let me know if I have this concept correct.
I would appreciate all feed back.

Thanks,
James


Thread source:: http://www.webmasterworld.com/robots_txt/4156501.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com