Often another site is linking to you using the IP address - this used to happen more often in the early days of the web (I know of one midwestern newspaper that did it regularly!).
But the exact reason in any particular case is not nearly as important as how to fix it. Essentially it is another kind of canonical problem. On your server, you can 301 redirect any request that uses the IP address to the same file but using the domain name. You can also include the canonical link element and make sure it uses the preferred address for the file.