Depends on where it appears. If it's just once in a while, it's probably not a big deal. If it happens a lot and a lot of money is getting yanked back, then you want to start looking at your traffic to make sure it's targeted and correct, and also maybe look at your placement to ensure that your ads aren't too close to your navigation or can't be mistaken for it.
You may need to inspect your logs to see where your traffic is coming from. For example, if I have a site targeted to one state in the US and suddenly I start getting a bunch of traffic from Russia that is clicking on ads, then there's obviously something wrong there, no matter how the traffic came to be there.
And if you do identify some kind of anomaly, consider removing the ads from the affected url(s) until you fix the problem.
If you can't fix the problem, then it's probably a good idea to look for something other than AdSense to monetize your site.