Welcome to WebmasterWorld Guest from 54.242.165.26

Forum Moderators: mack

Message Too Old, No Replies

Duplicate content indexed. Mod re-write and the robots.txt file. Safe?

     
12:03 am on Apr 25, 2008 (gmt 0)

New User

5+ Year Member

joined:Mar 20, 2008
posts: 21
votes: 0


I been getting double index in some files....
one with the Friendly URL and one with the Raw url

Raw format
www.example.com/page.php?page=profile&recid=acme
www.example.com/page.php?page=profile&recid=acmeplus

Friendly format
www.example.com/profile/acme.html
www.example.com/profile/acmeplus.html

Can I block the RAW Url on robots.txt. Is this safe?

User-agent: *
Disallow: /page.php

1:29 am on Apr 25, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


It would be better to find out *why* you're getting those "raw" URLs indexed, and to put a stop to it first.

Then 301-redirect any direct client requests only from the "raw" to the friendly URLs. I emphasized that phrase because it's important: You'll need to look at the client HTTP request server variable to be able to distiguish between a raw URL request received from a client (browser or 'bot) and a raw URL request resulting from your existing raw-to-friendly-URL internal rewrite.

But first, make sure you have no links on your site to these raw URLs, and try to get any such links from other Web sites that link to your site corrected before proceeding.

Jim

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members