I created the following Robots.txt file and want to block access to URLs ending in a certain pattern.
The URLs I want to block look like this:
http: //www.XYZ.com/-white/car.html
http: //www.XYZ.com/-blue/car.html
I dont want to block any URL that looks like this:
http: //www.XYZ.com/white-car.html
http: //www.XYZ.com/blue-car.html
so I have my robots.txt file as follows:
User-agent: *
Disallow: /*/car.html$
So this should only block URLs ending with " /car.html "
I want to make sure it only blocks URLs that have a " / " right before the "car.html" and not ones ending like" -car.html"
Can you guys let me know if I have this concept correct.
I would appreciate all feed back.
Thanks,
James