There's only really a small number places that malformed URLs can come from.
1. For certain malformed request X, the pages of your site link to malformed URLs due to an error in the scripts that run your site. Xenu LinkSleuth can prove or disprove this very quickly.
2. An error in the server configuration means that for certain requests the server returns a redirect to a malformed URL. Xenu is of use here too.
3. You're using internal links in the site navigation in the form href="..\..\..\somepage.html" and some bot is misinterpreting them in some way.
4. There's some other site linking to malformed URLs due to either an error in the scripting of the other site or else a simple cut and paste error by someone posting links there.
5. A bot has a bug such that it requests non-valid URLs from your site, even when presented with valid links.