Forum Moderators: open
What I'm interested in looking into is the possibility of serving a stripped-down version of each topic when a spider is detected. Identical content, but without the extra display html, Javascript, etc.
Would this be regarded as 'cloaking' in the wider-sense of 'abusing the spiders' or is this okay. Thinking mainly of Google's stance on this.
Haven't done anything yet, but don't want to jeopardise what we've achieved so far, though I don't see anything wrong with the 'stripped down' approach I'm thinking of.
Has anyone any experience of this?
Ta,
John
However there are always those who despise any form of cloaking.
Volatilegx, do you think there is risk in doing useragent based stripping of extraneous eye candy type stuff from pages? Without of course ever adding or altering any content on the page, the bot just gets a little less of the page that is. To me it seems reasonable, since the stuff stripped would just be repeated junk that has nothing to do with what the searcher is looking for. In other words, what Brett called 'benign cloaking' in another thread in this forum. I know sites like this use that, and other large sites, but I also suspect large sites can get away with more than smaller sites.
John
Volatilegx, do you think there is risk in doing useragent based stripping...
Sure there is a risk involved. Just where does Google draw the line? There isn't a definitive answer to this question. I think it is done on a case-by-case basis, but they claim it is automated (I doubt it -- the application of penalties is too erratic).
It is up to you to decide if the risk outweighs the reward. Personally, I think the risk is miniscule. BUT, if the site in question is very important to you, then you might not want to expose it to unnecessary risks.
Dan
Would this be regarded as 'cloaking' in the wider-sense of 'abusing the spiders' or is this okay.
I am inclined to think that its not what humans think about it that matters, but what robots can possibly think about it. Think like a robot:
1) you get a page with robot's useragent and calculate checksum X
2) you get same page with IE's useragent from previously unknown IP -- checksum is Y.
Note that checksum might not necesserily be straightforward MD5, but a better form that can take into account small changes in content (like current time that some sites show).
So, our robot detects that same page has different content depending on useragent/IP that was supplied. What can it possibly think about? Robots will not be as smart as humans for a long time, so the best strategy is to err on side of caution and consider that page suspicious and just ban it outright.
Cloaking is not a cheat in a number of cases (like when user logs in and get personalised content), but how can you be sure that robots will be able to detect something like that? They can't and in this world full of search engine spam its best to be safe than sorry.
