Welcome to WebmasterWorld Guest from 54.158.167.59

Forum Moderators: goodroi

Message Too Old, No Replies

Correctly disallowing only parts of a folder

   
10:14 am on Aug 13, 2009 (gmt 0)

10+ Year Member



In my robots.txt I copied a line from another site that i thought applied to mine as well because the idea is the same

I have a dynamic page that has 100s of links looking like this:
.com/widget-something/?category=something1&page=&text=0
.com/widget-something/?category=something2&page=&text=0
...

Now i ONLY want .com/widget-something/ to show in google
when linking to .com/widget-something/?category=something1&page=&text=0 i want the link juicy just to go towards .com/widget-something/ and not the specific URL

So i added this to the robots.txt

User-agent: *
Disallow: /widget-something/?

Is that wrong? Google wembaster tools says I am blocking /widget-something/

What am I doing wrong? How should it be done correctly?

9:50 pm on Aug 16, 2009 (gmt 0)

5+ Year Member



You could always config your webserver to process # symbol as a ? and this way crawlers will treat your variations:

.com/widget-something/#category=something1&page=&text=0
.com/widget-something/#category=something2&page=&text=0

as the single entity of:

.com/widget-something/

 

Featured Threads

Hot Threads This Week

Hot Threads This Month