Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

When to use noindex, 301 or robots.txt ?

         

realmaverick

9:11 pm on Mar 11, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Good evening guys, I'm working on clearing the tens of thousands of errors in webmaster tools, caused by various things. As well as weeding out "junk".

1. The forum software we use, utilises tags, that in the latest version, decided to start using friendly URL's. This means the tens of thousands of old tags, all now 404. In this case, I am going to do a 301, though I cannot get it to work. I've tried about a billion different things including:

RewriteRule ^(.*)\-tag\.html&tagtype=contentType$ http://www.example.com/content/contentType/tag/$1/ [R=301,L]
*FIXED THIS PART*

Old tag url is example.com/example-tag,html&tagtype=contentType to new tag http://www.example.com/content/contentType/tag/example/

2. We have thousands of 404's from a developers error, the 404 is correct, the pages don't exist and never will. The error is www.example.com/app=core/example... which I have Disallowed the entire directory in the robots.txt. The error in the code, which caused the links to the pages is also fixed.

3. 2 million profiles, many of which are identical. I have added noindex, follow to all profile pages as well as removed many of the links, that lead to profiles.

4. We went from using Wordpress blog to invision blog, as it integrates properly with our forum and the rest of our CMS. However the URL's don't match. They contain an ID, making a Rewrite rule impossible. My developer wrote a script, that contains 1,000 301's for the .htaccess. I'm hoping 1,000 lines of code, won't hinder performance.

Have I made the right choices?

[edited by: realmaverick at 10:07 pm (utc) on Mar 11, 2012]

realmaverick

8:14 pm on Mar 15, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks.

One issue I am unsure what to do, is the server returning server 500 errors, for pages that don't exist, that never existed but had links to them, from an error in code.

I have instead returned a 410 gone. But not sure that is ideal? I don't want to create a soft 404 problem. But I could perhaps redirect each one, to their nearest match?

g1smd

8:32 pm on Mar 15, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Find the error in the code and fix it rather than adding another layer of bandaids.

The Live HTTP Headers extension for Firefox should be used to investigate the server responses.

It is likely there is either an infinite loop for those requests or a simple pattern or syntax error in one particular rule.
This 32 message thread spans 2 pages: 32