homepage Welcome to WebmasterWorld Guest from 54.226.235.222
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
disallowing .asp pages
disallow asp pages
xert




msg:4240226
 6:51 pm on Dec 8, 2010 (gmt 0)

Hi,

I have an .asp page which I wish to block as google is listing it as a soft 404 with duplicate content. The page is referenced for example as :

review_form.asp?model1

review_form.asp?model2

review_form.asp?model3..... and so on.

Each page is being flagged by google as duplicate so I wish to block robots from accessing this page in its entirety.

I've searched for info on these boards and am not sure if i should be using an asterisk after the "?"

I'm currently using the following robots file :

User-agent: *
Disallow: /review_form.asp?

Is this correct, or should I be using something like :

User-agent: *
Disallow: /review_form.asp?*

Any help will be appreciated.

Regards to all.

Chris

 

phranque




msg:4240453
 9:28 am on Dec 9, 2010 (gmt 0)

the robots.txt pattern matches left-to-right, so your current usage is sufficient.

however, excluding a url with robots.txt will not prevent the snippetless url from being indexed nor from collecting PR.

depending on your particular application it may be better for you if you either use a meta robots noindex or 301 redirect to the canonical url.

xert




msg:4240589
 4:25 pm on Dec 9, 2010 (gmt 0)

Many thanks phranque - I've used the first robots file and seen the number of duplicate files listed in google tools reduced from 180 to around 120 so it would appear the code is working fine.

Best regards

Chris

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved