Welcome to WebmasterWorld Guest from 54.234.38.8

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt wildcard syntax?

is this syntax ok?

     
3:49 pm on May 31, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 20, 2004
posts: 68
votes: 0


I have thousands of these similar url's (see below) that needs to be eliminated from Google index, to avoid duplicate content.

/category6_7/product23/product_info.html
/category2_10/product636/product_info.html
/category77_33/product1171/product_info.html
/category23_35/product705/product_info.html


Is this the correct syntax for robots.txt?

Disallow: /category*/product*/product_info.html


I want to make sure before I make it live.

thanks for your help in advance..
8:00 pm on May 31, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


It might be.

Disallow: /*/product_info.html

OR
Disallow: /category*/product_info.html


might also be right.
8:26 pm on May 31, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 20, 2004
posts: 68
votes: 0


I'm trying to eliminate duplicate content with these "good" url's:

/product23/product_info.html
/product636/product_info.html
/product1171/product_info.html
/product705/product_info.html

so I want to be very careful with my wildcards.. ;-)
8:51 pm on May 31, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


I'd use the last one:

Disallow: /category*/product_info.html
9:39 pm on May 31, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 20, 2004
posts: 68
votes: 0


Thanks for your help..