So some of my pages from vbulletin require a login to see the content. For example, this might include user profile pages, search results pages, or private forum posts. Would it be correct to show a 404 to google, but 200 to logged in users in these cases? (so as to avoid the soft 404 problem?)
Msg#: 4359216 posted 2:07 am on Sep 7, 2011 (gmt 0)
the 401 is for Basic or Digest authentication and "The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource." http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2
Msg#: 4359216 posted 3:21 am on Sep 7, 2011 (gmt 0)
All-purpose rule: Do not give Google a 404 if there is any alternative whatsoever. It will keep looking for the page forever. Or until the robot's warranty expires, whichever comes first.
If you can, tuck the relevant pages into separate, roboted-out directories. Any links to them should be no-followed. (Belt and suspenders principle. Belt and braces, for the Brits.) And then, to be safe, fire up the htaccess and block google from seeing the pages at all.
But first: if you do nothing, what does Google get? Are the pages set up so it's physically impossible to reach them if you're not logged in? If so, google should already be getting a 403.