Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: goodroi

Message Too Old, No Replies

Blocked Shot - Robots.txt

blocking numerous duplicate pages using robots.txt

     
12:32 am on Oct 24, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:May 8, 2007
posts: 62
votes: 0


Can someone answer this for me? I need help. I have duplicate title problems and I need to block them using a robots.txt.

www.mystore.com/blue-widgets/2/

and then there are up to 142 other titles just like it.

How can I block or disallow this in robots.txt without blocking the directory/files and without having to use every number to up to 142?

Thank you!

3:05 am on Oct 24, 2009 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10544
votes: 8


the robots.txt matching is left-to-right, so if you
Disallow: /blue-widgets/
that will exclude anything in the blue-widgets directory.
note that you don't specify the domain in robots.txt.
the pattern matching is limited so i'm not sure how you could specify a range of numbers.

if those urls are already indexed, blocking with robots.txt may not really solve your duplicate content problem nor will it prevent your url from being indexed in the future.
the proper way to solve this is probably to use 301 redirects to the canonical url.

2:00 pm on Oct 24, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:May 8, 2007
posts:62
votes: 0


Thank you, phranque.

What if you want /bluewidgets/ but want to block /bluewidgets/2/, /bluewidgets/3/, /bluewidgets/4/ and so on?

They have not been indexed, so far not yet.

2:41 pm on Oct 25, 2009 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10544
votes: 8


Disallow: /blue-widgets/0
Disallow: /blue-widgets/1
Disallow: /blue-widgets/2
Disallow: /blue-widgets/3
Disallow: /blue-widgets/4
Disallow: /blue-widgets/5
Disallow: /blue-widgets/6
Disallow: /blue-widgets/7
Disallow: /blue-widgets/8
Disallow: /blue-widgets/9
4:30 pm on Oct 25, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:May 8, 2007
posts: 62
votes: 0


My Worst Fears! Yikes. Good Halloween Scare! Just have to take my lumps with Google.

Thank you, phranque!

10:50 am on Oct 27, 2009 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10544
votes: 8


it might not be so bad.

since it matches left-to-right you only need those 10 rules to exclude all subdirectories of blue-widgets that begin with a numeric and their contents.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members