Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Correctly disallowing only parts of a folder

10:14 am on Aug 13, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 30, 2003
votes: 0

In my robots.txt I copied a line from another site that i thought applied to mine as well because the idea is the same

I have a dynamic page that has 100s of links looking like this:

Now i ONLY want .com/widget-something/ to show in google
when linking to .com/widget-something/?category=something1&page=&text=0 i want the link juicy just to go towards .com/widget-something/ and not the specific URL

So i added this to the robots.txt

User-agent: *
Disallow: /widget-something/?

Is that wrong? Google wembaster tools says I am blocking /widget-something/

What am I doing wrong? How should it be done correctly?

9:50 pm on Aug 16, 2009 (gmt 0)

New User

10+ Year Member

joined:July 30, 2009
votes: 0

You could always config your webserver to process # symbol as a ? and this way crawlers will treat your variations:


as the single entity of: