TheMadScientist - 2:10 am on Mar 15, 2010 (gmt 0)

I kind of see this the way duplicate content filtering would deceive webmasters into think a penalty existed , when all it was only a filter at work.

It might not even be a filter, but rather the number of redirects they set their bot to follow... I'm working on one right now and will probably only follow one or two redirects at the most so if there are 3 in place I wouldn't know what links actually point to the final landing page, because it would be disconnected from the original since the number of followed redirects is limited.

It could be they follow 2, then request the second redirected to location and then follow 2 more and so on (or something to that effect) and then try to piece together what goes where, but I would personally try to get people to the correct location with 1 or 2 redirects at the most rather than relying on an association being made and not having the link weight severely discounted, if counted at all.

How and what to follow how far is actually one of the questions I was trying to figure out the answer to today, so right now it's only a theory and something I'm trying to work through myself for a 'niche specific' bot.

What they do is probably a way more complicated than I'm making it sound (what I'm doing is), but I could see where there may be a 'disconnect' of links to landing page if there are too many redirects in place, or why you would want to disconnect the links to one page from the final landing page if they run through too many redirects... It's one of those things that's a bit tough to explain and easier to understand if you sit down and try to do it.

Here's one way to look at it:
You get the info from Page A, which has a link to Page B and it's fairly easy to store and associate the fact A links to B, but then you visit Page B and it redirects to Page C, so you have to change the association of links somehow from A links to B and say A links to C, then you open up Page C and find it redirects to Page D, so you have to redo the association again to say A links to D somehow, and you either have to drop the redirected pages out of the middle, or find a way to associate the links through the redirects and it turns into a really complicated question as far as storage, association and access are concerned, especially when you start asking yourself what if Page B no longer redirects to C, but rather to F?

It's not a simple as a browser redirecting, because a browser simply takes you there and if the redirects change between your visits it doesn't really matter your browser takes you to the new location instead, but it doesn't work that way in a database...

[edited by: TheMadScientist at 2:45 am (utc) on Mar 15, 2010]