TheMadScientist - 2:11 pm on Dec 12, 2012 (gmt 0)
Very few robots take the trouble to read your 404 page. They just note the 404 and go on their way.
Where do you get that?
The only way they don't 'take the trouble to read the page' is if they request it using a HEAD request, but we already know Google has explicitly stated they don't use a HEAD request because it really doesn't save much on resources or speed requesting up, so they want to know what's on the pages they request and use GET.
I also don't see a large number of HEAD requests for 404 (or any) pages from bots, so where exactly are you getting your info, because they DO take the trouble to get the whole page if they use GET, which means unless you have access to every engine's handling algo, you cannot know exactly what any given engine does or does not do with any given page or info based on status code, unless you have an explicit statement from someone who actually works at each engine you're referring to as part of very few (and all the rest you say 'note it and go on their way') and I haven't found those statements anywhere yet.
So, please, if you have that type of info from search engines, cite your source(s). Thanks.