I don't have a smart phone, but was able to reproduce the desired styles on a PC. However, as I'm not testing on the correct device, I've laid out my reasoning so you can decide which bits help. If any do, let me know and I'll make a broader call to see if we can find some people with the right devices to test.
Media queries are in flux as the candidate recommendation is being revised but is still in editors draft, and they are affected by HTML5 which is still a working draft. However, best I can tell, the following has not changed:
The ‘width’ media feature describes the width of the targeted display area of the output device. For continuous media, this is the width of the viewport ... including the size of a rendered scroll bar (if any). and: ‘device-width’ media feature describes the width of the rendering surface of the output device. For continuous media, this is the width of the screen. Media Queries [w3.org] (My bold)
Reducing this for simplicity, your css is querying: (minimum device rendering width:320px) and (maximum device rendering width:480px) =background-color:green and (maximum viewport width:320px) = background-color:yellow
That means the queries are measuring two different things, and the first rule (green background) won't apply if the device has a width greater than 480px (such as a PC monitor), but the second rule (yellow background) will apply on any device when the viewport is resized below 320px.
That allowed me to reproduce all the desired style changes* in ff and Op, plus winSafari if a greater width was used (because the PC version won't allow the viewport to be reduced below around 328px). ie<9 do not support media queries, but I'd expect the same behaviour in ie9. Just to test I also replaced max-device-width with max-width and the first rule (background-color:green) was also applied as the viewport resized.
That might indicate width would be a better option than device-width, but I'd suggest more research: Some devices have issues reporting width and that may or may not affect site performance in the wild.
* On font-size, that is set at 20% - were you really wanting it that small?