Is it the physical size or the pixel count that changes? If you're trying it on different computers-- which is a terrific idea precisely because it's showing you things that don't work as you'd hoped-- you may be looking at not just different fonts* but different dpi/ppi in the display. So you've got three different measurements: ems, physical size (inches or cm) and pixel count. Does any aspect of the page change if you change the monitor's resolution but nothing else?
I once spent an exhausting session trying to make my son understand that in modern monitors a pixel is not only a physical dot on the screen, it's also a unit of measurement. The first generation of Mac and Windows were 72 and 96dpi respectively. Those were easy to understand, because the same number meant both; they were actual points of light that you could see if you looked close enough. But now the measurement and the physical dot are separate and unrelated things.
:: shuffling papers ::
[lucysworlds.com...] (531byte gif image)
At the time I made this image (for use with e-texts using words like "actual size") in 2007, the ruler displayed at actual size on my monitor. Now, on a different computer, it's a bit smaller.
* That is, even if you say explicitly "14 point Palatino", different computers may render it differently depending on age of the monitor and source of the font.