Yes, this is worrying on servers. My linux server does the same thing. PC folks are used to having lots of memory available, and running out is a bad thing. Servers, as robzilla points out, are different.
I would check CPU load as well - that's likely to be much more telling about what's going on on the server. You do want lots of CPU load available. Mine runs at a few percentage points of capacity - much higher than that and I'd start getting concerned.