Welcome to WebmasterWorld Guest from 54.145.65.62

Forum Moderators: goodroi

Message Too Old, No Replies

Syntax for disallowing folder with '.' in the name?

Asking for clarification

     
9:36 pm on Jun 23, 2008 (gmt 0)

Full Member

10+ Year Member

joined:Apr 1, 2003
posts:267
votes: 0


Hey there,

I know virtually nothing about robots.txt so forgive me if this question is silly. I figured better safe than sorry. Anyway...

For whatever reason this site has it's directory structure setup like this:

http://www.example.com/product.php/0001
http://www.example.com/product.php/0002
http://www.example.com/product.php/0003
etc.

And:

http://www.example.com/company.php/1234
http://www.example.com/company.php/1235
http://www.example.com/company.php/1236
etc.

They want to block only the /company.php/* ones for fear of overlapping duplicate content (many of the companies only have only 1 product and thus the pages are virtually identical).

I was just wondering what the correct disallow syntax is for that? Thanks!

9:57 pm on June 23, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


User-agent: *
Disallow: /compan

Put as much or as little of the /company.php/.... part into the Disallow statement as you like, enough to make it globally unique.

10:40 pm on June 23, 2008 (gmt 0)

Full Member

10+ Year Member

joined:Apr 1, 2003
posts:267
votes: 0


Great, thanks g1smd!