homepage Welcome to WebmasterWorld Guest from 54.237.54.83
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Correctly disallowing only parts of a folder
Trax




msg:3971084
 10:14 am on Aug 13, 2009 (gmt 0)

In my robots.txt I copied a line from another site that i thought applied to mine as well because the idea is the same

I have a dynamic page that has 100s of links looking like this:
.com/widget-something/?category=something1&page=&text=0
.com/widget-something/?category=something2&page=&text=0
...

Now i ONLY want .com/widget-something/ to show in google
when linking to .com/widget-something/?category=something1&page=&text=0 i want the link juicy just to go towards .com/widget-something/ and not the specific URL

So i added this to the robots.txt

User-agent: *
Disallow: /widget-something/?

Is that wrong? Google wembaster tools says I am blocking /widget-something/

What am I doing wrong? How should it be done correctly?

 

echwa




msg:3973043
 9:50 pm on Aug 16, 2009 (gmt 0)

You could always config your webserver to process # symbol as a ? and this way crawlers will treat your variations:

.com/widget-something/#category=something1&page=&text=0
.com/widget-something/#category=something2&page=&text=0

as the single entity of:

.com/widget-something/

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved