SevenCubed - 6:36 pm on Feb 9, 2011 (gmt 0)
At the very root of my post is that I am saying html validation is one of the surest ways to gain in SERPs. I have taken it further by sharing some of my observations that resulted from validation with a twist. I would not have stated what I have if it was not tested and true. Whether or not Google gives kudos to those back links is up for each individual to decide for themselves. I know it has been beneficial for me, your results may differ.
But, it can be reproduced by everyone and you will see indexed results. Beyond that, interpret it as you want. Regardless of what W3C's robots.txt file says, and I wasn't aware of that, if applied as I have outlined you will get pages indexed by Google from W3C (apparently also non-valid ones which I also wasn't aware of -- you're right frank72 and McMohan). Why Google ignores the do-not-crawl directive from W3C or how they do it is not my concern. I'm simply saying it is so. Nobody was more surprised than me to see back linking to my site indexed in Google SERPs from W3C than me. But I did specifically implement it in that manner to see if it could be done and much to my delite it was.
For those who are saying that we shouldn't be concerned about the technical stuff and just do what's best for the visitor -- I agree. But for me, high on a good user experience list is the technical stuff, unknown and unapparent to them. Valid html and css, and even the order of recursion of it if you really want to dig down even deeper, makes a page lightning fast to render and load. I could go into even more technical detail but that would lead me too far off topic.
And like netmeg mentioned -- information architecture is also right up there at the top. My mind functions optimally in images and patterns and I build sites, for lack of a way of explaining it very well, in binary form for the search engines (it's a story within a story) and they also serve as a dual meaningful visual presentation for visitors. Often times I cannot easily express in words what I am processing internally. But the sites I work on are produced in patterns not words. The written words become the result of the underlying patterns I have applied. And, most of the time, they are very effective. For me, technical stuff is most important, that is SEO from my way of thinking. But that technicality is not apparent to the reader, based on analytics, the sites are sticky so it must be appealing to them or answering their questions.
Maybe pageoneresults is right. Maybe because of this thread exposing this ability something will be done by someone to prevent it from continuing but even if it does it makes no matter to me because my pages still validate and that's what's most important. I have no doubt that Google assigns a healthy portion of their algorithm to on site performance factors. Those are the things I focus on because they are within my control rather than off-site factors that are not.