homepage Welcome to WebmasterWorld Guest from 204.236.254.124
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

    
Duplicate content indexed. Mod re-write and the robots.txt file. Safe?
rudyten




msg:3634626
 12:03 am on Apr 25, 2008 (gmt 0)

I been getting double index in some files....
one with the Friendly URL and one with the Raw url

Raw format
www.example.com/page.php?page=profile&recid=acme
www.example.com/page.php?page=profile&recid=acmeplus

Friendly format
www.example.com/profile/acme.html
www.example.com/profile/acmeplus.html

Can I block the RAW Url on robots.txt. Is this safe?

User-agent: *
Disallow: /page.php

 

jdMorgan




msg:3634670
 1:29 am on Apr 25, 2008 (gmt 0)

It would be better to find out *why* you're getting those "raw" URLs indexed, and to put a stop to it first.

Then 301-redirect any direct client requests only from the "raw" to the friendly URLs. I emphasized that phrase because it's important: You'll need to look at the client HTTP request server variable to be able to distiguish between a raw URL request received from a client (browser or 'bot) and a raw URL request resulting from your existing raw-to-friendly-URL internal rewrite.

But first, make sure you have no links on your site to these raw URLs, and try to get any such links from other Web sites that link to your site corrected before proceeding.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved