brotherhood_of_LAN - 5:04 pm on Dec 27, 2012 (gmt 0)

Traditionally the issue was a session parameter in the query string that would normally be also served in a cookie.

Because the spiders did not perform requests with cookie's, they would get issued a new session for every page load- essentially this meant an almost infinite number of unique URLs, and also trapping the spider and giving them lots of duplicate content under those URLs.

There was possibly a time when people thought URL rewriting was good to give the impression your site was 'static' and not served by a scripting language.... but I think in both of these cases are obsolete.