TheMadScientist - 10:40 pm on Feb 8, 2010 (gmt 0)
LMAO... Yeah, you'd think it would be simple.
Why is it so complicated? Google's way of saying thanks for letting us spider your site and have access to your content in the first place... If you want us to let go of it you'll have to work for it. LOL.
If you remove the robots.txt block and set up a custom 404 page with the meta tag below they won't return the cache or the pages in the results any more, but you will have to wait for them to spider the pages to get it to take effect. It is the simplest, most straight forward way IMO.
Also, for future readers, although you should be able to serve a 410 Gone as outlined above, know if your pages are ever really missing 404 you will be serving a 410 instead, so it's not something to use, unless you know for sure you want all missing (404) pages dropped and not spidered for a longer period of time.
All you really need is to get the SEs the following and all references to the page it's on will be dropped from the results. Personally I almost always serve noarchive on my pages, even when I allow the pages to be indexed and returned in the results.
<meta name="robots" content="noindex,nofollow,noarchive">