homepage Welcome to WebmasterWorld Guest from 54.235.16.159
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt wildcard syntax?
is this syntax ok?
maha




msg:4320022
 3:49 pm on May 31, 2011 (gmt 0)

I have thousands of these similar url's (see below) that needs to be eliminated from Google index, to avoid duplicate content.

/category6_7/product23/product_info.html
/category2_10/product636/product_info.html
/category77_33/product1171/product_info.html
/category23_35/product705/product_info.html


Is this the correct syntax for robots.txt?

Disallow: /category*/product*/product_info.html


I want to make sure before I make it live.

thanks for your help in advance..

 

g1smd




msg:4320170
 8:00 pm on May 31, 2011 (gmt 0)

It might be.

Disallow: /*/product_info.html
OR
Disallow: /category*/product_info.html

might also be right.

maha




msg:4320185
 8:26 pm on May 31, 2011 (gmt 0)

I'm trying to eliminate duplicate content with these "good" url's:

/product23/product_info.html
/product636/product_info.html
/product1171/product_info.html
/product705/product_info.html

so I want to be very careful with my wildcards.. ;-)

g1smd




msg:4320193
 8:51 pm on May 31, 2011 (gmt 0)

I'd use the last one:

Disallow: /category*/product_info.html
maha




msg:4320218
 9:39 pm on May 31, 2011 (gmt 0)

Thanks for your help..

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved