TheOptimizationIdiot - 2:46 am on Apr 3, 2013 (gmt 0)
There's actually a bunch more that goes into the decision making when you're dealing with 1,000,000,000,000+ URIs and running a business.
First, there are so many who thought .html was better than .php (and may still be) eliminating .html with a parameter would be silly since people thought they would do better by parsing .htm and .html pages as php and using parameters on them.
Third, when you run a major search engine and have the insane number of pages and URIs they have to deal with you hit a point of diminishing returns by worrying about coding for minutia like newwindow=true is, so what's way more cost effective than trying to figure out all the parameters you don't need to crawl is to spider the URI and see if it returns a 200 OK header, then if it does you do what you do (as Google does) and group the URIs with the same content together and 'give value to/return in the actual SERPs' what you determine to be 'the best/most authoritative one' you find.
When you really get into running a search engine and trying to figure out what to do with 1,000,000,000,000+ URIs/pages there's really a bunch of reasons to spider and 'let slide' a bunch of things many of us who don't deal with those numbers might think are silly or easy to fix, but they're really not 'that important' when you deal with things on the scale they have to code for, especially when you get into how time consuming finding and coding solutions for some of the things must be and how much better that time finding and coding for issues could be spent doing something else.
(For example, not picking on you Lucy24, I wouldn't have ever thought about coding for newwindow=true and with the number of URIs they have to deal with they might not have a clue someone was silly enough to even use it, so is the time spent digging through the insane number of URIs they have to deal with to find the 'goofiness' some people erroneously link with worth the time invested when they'll probably find a (relative) few at most or is the time of some search engineer with a doctorate and $1,000 an hour salary probably better spent somewhere else? IOW: how much would they profit by "eliminating" newwindow=true from the index and how is that possibly more than they would spend by finding and coding for it and other silliness on the part of webmasters? I don't see how they could really be bothered with it personally.)