Recently, a new wave of Web morality has decreed cloaking to be spam, and yet the practice is increasing not decreasing, for all the warnings about getting banned and/or penalised more sites seem to drop that have done nothing more than put HTML to the page than those that do some really esoteric stuff.
Why is this? Because more and more sites are serving different content to not only spiders, but to browsers being driven by real human beings as well, which version did the spider see? Search engines themselves routinely serve their customers different pages depending on browser variables. Of course that does not justify anything, it simply says that serving different (formatted) content is desireable under certain circumstances and that spiders routinely get different content from what you and I might see.
So that takes us back to why do sites continue to rank well if they are cloaking? If it is such a sure fire way to get banned, or buried so deep that a GPS couldn't locate you, they should all be gone, right?
Well they are not gone because IMO search engines don't care about cloaking, it isn't a big deal, never was, "spam" (if anyone could truly define it) is a big deal, inappropriate content switching is a big deal, cloaking isn't. It ranks right up there with begging for links and paying review fees to directories that for most sites provide no direct traffic.
It could be argued that the lack of defining what IS acceptable by the Search Engines provides an even greater impetus to selectively deliver pages. One SE's optimized and relevant page is another's spam page. The real target of search engines is not morality or fairness, if they don't like optimization [SEO, doorways, cloaking, position checks, index spidering, near duplicate pages, and so on] it's becasue it has the potential to homogenize the search results across a number of SE's, not because they want to level the playing field. What is the differentiating factor for a Search Engine if they all have the same 30 pages as their top results? Yet that is what SEO's consistently strive to do, whether you are a dyed in the wool spammer or a content is king sort of guy or gal.
In a world where Yahoo serves me pop under windows, AltaVista maximizes windows automatically, Google caches my pages, AOL won't let me keep decent stats, Microsoft prevents my browser from seeing their site, my 404 traffic is hijacked by IE, search engines serve me different results by browser and location, Inktomi turfs pages to encourage more PPC, Looksmart mangles descriptions and then charges to fix them, Search Engine spiders chew bandwidth without indexing, I alone must walk the ethical line? What in the world could that be?