Forum Modification Concerns

[Asia_Expat]

I've done a really simple edit on my IPB forum installation that means it now produces a 410 server response for all error pages. This fixes the issue of old removed topics being indexed with the boilerplate error message in IPB.

However, I am a little concerned about one this... the 410 response is also produced for things parts of the forum unavailable to guests, such as the member profiles and of course, member profile links are in every topic. Could having links on every page to 410 pages cause me issues?

Incidentally, member profile URL's are also blocked in the robots.txt file.

[edited by: caveman at 7:23 pm (utc) on Jan. 24, 2008]
[edit reason] Removed code [/edit]

[caveman]

One obvious, and negative, implication is that in so doing, you are bleeding valuable link equity into tons of little black holes. Not good.

You could "nofollow" all those links, which would stop the SE's from going down those paths. You want to be sure that you add nofollow to every single link that leads to a 410.

[Asia_Expat]

So, is blocking via robots.txt insufficient?

[caveman]

Correct. Link equity could still flow into those holes.

[Asia_Expat]

... and nofollow on those links is a cure all, and will not flag my own site as one that I do not trust?

[caveman]

Ugh. How many of these are there on the site, let's say, as a percent of total internal links?

I don't think nofollow is a panecea at all, in fact I hate recommending it, mainly because I think it was conceived as a way of search engines getting Webmasters to do their work for them, and because I don't like how they evolved its use into other areas.

I have speculated before that it would make sense for the SE's to red flag sites that seem to be using nofollow in highly manipulative ways...and in my mind, looking at the percent of usage across all internal links might be one way of doing that. Looking at its use on internal versus external links might also be revealing. Whether any of that is actually being done I do not know.

A fair number of sites do use nofollow, and it does have the advantage of doing something that no other bot-related instruction can do, i.e., stop bot crawling at the link level. (You can also redirect through robots.txt protected directories, but that is more of a work-around than a bot instruction, and as far as I know, doesn't prevent wasted link juice.)

If your site has a ton of these links, that is obviously not good, and nofollow would be an inelegant solution versus creating a site structure that doesn't create the problem in the first place.