If you have a fairly recent motherboard, it should have one or more fan connector(s) that support varying the speed of the fan with temperature. (Typically this is NOT on all of the fan connectors.) Your BIOS should have some setup for this under "Power". If you have this feature, make sure you are using fans with 3-pin connectors that can use this feature. (The 3rd pin feeds a tachometer signal back to the motherboard.)
If you don't have such a motherboard, or if the feature doesn't work or work well for you, you can get fans with either an integral or remote temperature sensor. Make sure to plug these into a 3-pin socket that does NOT vary the fan speed with temperature (or turn this feature off in the BIOS) or connect the fan(s) through an adapter to a regular 4-pin "drive cable" power connector.
The idea is to spin the fans at a lower speed when your system is a idle and generating less heat - resulting in lower noise.
Larger dimeter fans are always a good idea. A larger diameter fan will move more air with less noise. (Because it can move more air at a lower RPM). You probably have no choice for your case and power supply fans (though many newer power supplies come with 120mm fans, so power supply replacement is an option.) But there are lots of 120mm options now available for CPU heatsink/fans.
To help you select fans, you can look at AF/N (airflow over noise) figures for fans. You will find that these are almost never published, but you can calculate it easily.
However, these vary a lot less than you would think. Most "low noise" fans are no different than any other fan - they just spin at a lower speed! (And provide less airflow.)
Popular 80mm fans range from 1.0 to 1.5 AF/N - so there is not much difference. Popular 120mm fans range from 2.0 to 3.375 AF/N, though most fall in the 2.0 to 2.5 range.
Finally, you can try to do some sound-proofing. The peel-and-stick open-cell foam typically sold for this purpose is nearly worthless. I'm going to to try a combination of 1/4" (2 lb/sq ft) MLV (mass loaded vinyl) and 3/8" closed-cell foam to further silence two rack-mounted computer cases and a rack case. I'll report back once I've done the installation.