Forum Moderators: goodroi
I know virtually nothing about robots.txt so forgive me if this question is silly. I figured better safe than sorry. Anyway...
For whatever reason this site has it's directory structure setup like this:
http://www.example.com/product.php/0001
http://www.example.com/product.php/0002
http://www.example.com/product.php/0003
etc.
And:
http://www.example.com/company.php/1234
http://www.example.com/company.php/1235
http://www.example.com/company.php/1236
etc.
They want to block only the /company.php/* ones for fear of overlapping duplicate content (many of the companies only have only 1 product and thus the pages are virtually identical).
I was just wondering what the correct disallow syntax is for that? Thanks!