| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor|
|Syntax for disallowing folder with '.' in the name?|
Asking for clarification
| 9:36 pm on Jun 23, 2008 (gmt 0)|
I know virtually nothing about robots.txt so forgive me if this question is silly. I figured better safe than sorry. Anyway...
For whatever reason this site has it's directory structure setup like this:
They want to block only the /company.php/* ones for fear of overlapping duplicate content (many of the companies only have only 1 product and thus the pages are virtually identical).
I was just wondering what the correct disallow syntax is for that? Thanks!
| 9:57 pm on Jun 23, 2008 (gmt 0)|
Put as much or as little of the /company.php/.... part into the Disallow statement as you like, enough to make it globally unique.
| 10:40 pm on Jun 23, 2008 (gmt 0)|
Great, thanks g1smd!
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved