homepage Welcome to WebmasterWorld Guest from 54.166.105.24
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Website Disallowed through robots.txt, will google re index
Website Disallowed through robots.txt, will google re index
jackgordon

5+ Year Member



 
Msg#: 3858711 posted 10:48 pm on Feb 26, 2009 (gmt 0)

We recently underwent some changes to our website, mainly url re-write.
It has been done in such a way that after a certain point, the rest of the URL does not matter.

For Example:..... the true page is "/glow_categories/41/glow_sticks.html" however our server knows to go to category "41",
so even "/glow_categories/41/webmasterworldqwerty.html" would still go to the same page. Make sense?

Anyway...
we had written some links on our website incorrectly, and they appeared as /glow_categories/41/glowsticks.html. They still go to the same page as the original link posted above.

The links have been updated however Google have already indexed them. If I exclude these through my robots.txt file to get them removed from their index, will Google re index them although they are not linked to on any of our pages?

Thanks

 

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3858711 posted 11:12 pm on Feb 26, 2009 (gmt 0)

Consider 301-redirecting the incorrect links to the correct ones. This will explicitly tell the search engines that those URLs are wrong, and that the corrected ones should be indexed instead. I suspect you might see a faster disappearance of the bad URLs using this method, and any page rank accrued by the bad links will pass to the correct ones through the 301.

It has been done in such a way that after a certain point, the rest of the URL does not matter.

This is essentially creating a duplicate-content problem for yourself, and one that could be exploited by competitors. You should add logic to your page-generation script that tests the URL PathInfo against your database, and issues a 301 redirect to the correct URL if the PathInfo does not contain the canonical URL-path for that "page."

Jim

schumi

5+ Year Member



 
Msg#: 3858711 posted 9:41 am on Feb 28, 2009 (gmt 0)

Jackgordon,

you will help SE decide which's the most matching url for the same content,so,just do it.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved