Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Correctly disallowing only parts of a folder



10:14 am on Aug 13, 2009 (gmt 0)

10+ Year Member

In my robots.txt I copied a line from another site that i thought applied to mine as well because the idea is the same

I have a dynamic page that has 100s of links looking like this:

Now i ONLY want .com/widget-something/ to show in google
when linking to .com/widget-something/?category=something1&page=&text=0 i want the link juicy just to go towards .com/widget-something/ and not the specific URL

So i added this to the robots.txt

User-agent: *
Disallow: /widget-something/?

Is that wrong? Google wembaster tools says I am blocking /widget-something/

What am I doing wrong? How should it be done correctly?


9:50 pm on Aug 16, 2009 (gmt 0)

5+ Year Member

You could always config your webserver to process # symbol as a ? and this way crawlers will treat your variations:


as the single entity of:



Featured Threads

Hot Threads This Week

Hot Threads This Month