If they are genuinely sending a forged IP, and it's landing in your raw logs that way, be afraid. Be very afraid. I'd always understood that this is the one thing robots-- or, for that matter, humans-- can't do. Much as they'd like to.
But nobody can be sure unless you post a sample from your raw logs. You imply in your first post that it's your own server, so it's not a question of access to the logs.