lucy24 - 2:39 am on Jan 22, 2012 (gmt 0)
Any php page (or jsp, asp etc.) will use a little bit of memory simply by building the page. The bandwidth kicks in at the next stage, when it sends the page-- along with any files such as stylesheets or images that the page asks for. Normally only humans get this extra stuff; robots concentrate on pages. More memory, less bandwidth compared to humans.
If a robot puts in a request for something with a query, then the php (or whatever) will use up a certain amount of memory trying to make sense of the query even if the php function ends up not being able to do anything with it. Similarly if the page requires a query and there isn't one, that will also take up memory. If you have a lot of this happening-- either fake queries or absent queries-- you need to intercept them in htaccess and block the request before it gets to the page. If a 403 is coming from the php page itself, that's memory being wasted.
A 406 error is very weird and probably wrong. It is definitely not supposed to mean "I don't like your face". That's what a 503 is for. A 406 is more like "I'm sorry, we don't carry that in your size".
And I have a ton of 404s for pages I deleted some months ago
Slightly off topic, but if a page used to exist and you deleted it on purpose, you shouldn't let it return a 404. Either redirect to a good replacement page, or use a 410 ("Gone"). A 410 will also get rid of robots faster than a 404.
long since I seen a log
Look at them periodically. You can get more information out of raw logs than out of fancy analytics. But you have to look in the right places. I've got a log-wrangling routine that does a lot of things semi-automatically. That includes getting rid of all image/css/js requests that belong to a page; and ignoring all authorized robots and all 403s. What's left tells you the two things you really need to know: what your human visitors are doing, and what kinds of unauthorized robots are snuffling around.