Cleaning that up a bit, we get:
RewriteCond %{HTTP_HOST} ^([^.]+\.)*app\.trunk\.?(:[0-9]+)?$
RewriteCond $1 !^(images|js)/[^.]*\.
RewriteCond $1 !\.(css|js|swf)$
RewriteCond $1 !^admin_dev\.php$
RewriteRule ^(.*)$ admin.php [L]
Note that I made the references to "admin_dev.php" and "admin.php" consistent. If either or both are in "/web", then this may not be correct. If this .htaccess code is in /web/.htaccess, then this code is not correct.
I made your HTTP_Host pattern much more specific, so it that cannot be 'spoofed' by using a subdomain like app.trunk.<anything-else>.example.com, for example. You may or may not want to make it case-insensitive by adding an [NC] flag. (Actually, you should have rules already in place ahead of this internal rewrite rule that externally redirect all non-canonical hostname requests to their respective canonical hostnames. If so, then you can use an exact match here, instead leaving the pattern un-anchored as in your original.
I combined and moved the "exclude directory" RewriteConds above the "exclude filetypes" RewriteCond, on the assumption that if most images and JS files are in those two directories, then there no need to check the filetype, and so a small gain in efficiency can be realized.
I also anchored all of these RewriteCond patterns as possible and escaped the literal periods, again for efficiency.
I removed your HTTP_REFERER RewriteCond. This is likely the crux of the problem, in that the HTTP Referer header is utterly unreliable. It will not be sent if the URL is typed in directly, accessed from a bookmark, accessed by a search engine robot, or accessed through any type of corporate or network proxy (e.g. AOL, EarthLink, etc.), and it will not be sent if the access is invoked by JavaScript. Therefore, it may be blank just when you need it the most...
Aside from my assumptions about the current and root directory loactions as implied by the start-anchors I added and the paths I adjusted, this code should work.
If not, then the whole concept fails, and you will need to do something else. For example, add something/anything to the
URLs requested by your AJAX so that they can be identified and disposed properly. This could be an alternate directory-path, a modified "filename" or "filetype," or a query string appended to the URL in order to identify it as a "not-to-be-rewritten" request.
Note the space between "!" and "admin_dev.php$" in the code you posted. If that was not a typo or copy-and-paste error, then it was a syntax error in the code and it should have caused a 500-Server Error. But if it didn't, then perhaps that's why your exclusion failed...
The [QSA] flag (Query String Append) is only needed if you want to *add* something to an existing query string. Otherwise, it's a waste of time and bytes.
Jim