MikeNoLastName - 12:11 pm on Sep 9, 2012 (gmt 0)

>Yes, they can "index" the URL - but they won't "crawl" its content and insert that content into the search results<

My experience seems to seriously contradict this. If you read my earlier post. They obviously DO crawl it's content and DO keep it in the internal database, since they were able to show my blocked page (without displaying the actual title and description) first in the results based primarily on it's title.

We can't control who links to a page from off our site which leads to another glaring example of how a competitor COULD affect your rankings despite G's disclaimers. IF they know you have a duplicate of pages of the site somewhere which is robot.txt disallowed. And IF G keeps all that disallowed content locked away in it's database (as it appears it does), but accidentally forgot to exclude it from their algorithm to judge duplicate content, then all a competitor needs to do is set some links to your disallowed duplicate content. It apparently doesn't even matter if they put any anchor text.