Welcome to WebmasterWorld Guest from 54.211.136.250

Forum Moderators: goodroi

Message Too Old, No Replies

Blocked Shot - Robots.txt

blocking numerous duplicate pages using robots.txt

   
12:32 am on Oct 24, 2009 (gmt 0)

5+ Year Member



Can someone answer this for me? I need help. I have duplicate title problems and I need to block them using a robots.txt.

www.mystore.com/blue-widgets/2/

and then there are up to 142 other titles just like it.

How can I block or disallow this in robots.txt without blocking the directory/files and without having to use every number to up to 142?

Thank you!

3:05 am on Oct 24, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



the robots.txt matching is left-to-right, so if you
Disallow: /blue-widgets/
that will exclude anything in the blue-widgets directory.
note that you don't specify the domain in robots.txt.
the pattern matching is limited so i'm not sure how you could specify a range of numbers.

if those urls are already indexed, blocking with robots.txt may not really solve your duplicate content problem nor will it prevent your url from being indexed in the future.
the proper way to solve this is probably to use 301 redirects to the canonical url.

2:00 pm on Oct 24, 2009 (gmt 0)

5+ Year Member



Thank you, phranque.

What if you want /bluewidgets/ but want to block /bluewidgets/2/, /bluewidgets/3/, /bluewidgets/4/ and so on?

They have not been indexed, so far not yet.

2:41 pm on Oct 25, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Disallow: /blue-widgets/0
Disallow: /blue-widgets/1
Disallow: /blue-widgets/2
Disallow: /blue-widgets/3
Disallow: /blue-widgets/4
Disallow: /blue-widgets/5
Disallow: /blue-widgets/6
Disallow: /blue-widgets/7
Disallow: /blue-widgets/8
Disallow: /blue-widgets/9
4:30 pm on Oct 25, 2009 (gmt 0)

5+ Year Member



My Worst Fears! Yikes. Good Halloween Scare! Just have to take my lumps with Google.

Thank you, phranque!

10:50 am on Oct 27, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



it might not be so bad.

since it matches left-to-right you only need those 10 rules to exclude all subdirectories of blue-widgets that begin with a numeric and their contents.