I don't know of any way to generate an ico file with PhotoShop... A piece of software called Icon Forge will apparently handle creating ico files, and if you're just looking to create an Internet Explorer favicon, favicon.com has an online icon generator you can use.
Here's an earlier thread on the subject, with some more links and info for you:
If you know the dimensions of your ico (16x16 or..?), you could set photoshop to indexed color mode, create a new file the correct size, enlarge the window to about 400% magnification, and draw the icon pixel-by-pixel with the pencil tool. You could then save it through Icon Forge and probably end up with exactly what you expected.
But if you take a regular psd file, save it as a gif, and then convert it to ico, you should probably prepare yourself for a lot of trial and error...