One thing that hackers do is find sites to help distribute malware. This one should be easy to detect, because Google will post a warning notice in the SERPs "This site may harm your computer." This discussion [webmasterworld.com] covers the details of how to handle a malware hack.
One common footprint for a malware hack is an iframe that doesn't belong in your code - especially one with a lot of hex coding.
These are really "old school" - they're more like online graffiti than anything else. The hacker usually just wants to brag that they got you, and they put up a message on your pages for all to see. Well, that's easily detected because you just go to your pages and there it is!
But as I said, this is old school and many hackers are looking for something with some financial value these days.
This one is either done for sheer malicious delight, or perhaps for competitive disruption. How often do you check your robots.txt file? If someone replaced the first line and disallowed all indexing, how fast could you catch that?
In addition to visually inspecting your robots.txt file on a regular basis (and especially if your urls start disappearing from the Google index) you can also set up a Webmaster Tools account and check it regularly. Google will report to you when urls get blocked by robots.txt.
This one is sneakier and depends on the value of backlinks, either for PageRank or for the traffic itself. The hacker places links on your pages (they may be hidden through various means) and you may not be inspecting your content close enough to see those links.
The tool you need is a link checker, such as Xenu LinkSleuth, that can give you a report on all your external links. You are careful about who you link out ot, right? So anything really bogus is going to jump out at you from that list. Running a link checker on a regular basis has many other benefits as well, such as keeping those accidental 404s out of your site. So I consider it to be something like getting a regular physical (but I recommend doing it more often.)
Now we're really getting devious. Over the past year or more, hacks have been showing up that cloak their parasite content so that only googlebot sees it. If you visit with a regular browser (user agent) you only see what you expected to see.
Your main tool here is a user-agent spoofer of your own, such as the User Agent Switcher extension for Firefox. Just fire it up with a googlebot user agent string and see if your page content changes.
Complex Cloaking - using IP and cookies
This is getting deep - and it's also not so common, but it is out there "in the wild." The hacker in this case paces complex scripting on your site so that not only do they cloak for googlebot by user agent, they also cloak by IP address. In some cases the script also places a cookie so you get only one chance to see what they're doing.
And your tools here are 1) learning how to browse your site with coolies turned off and 2) studying your server logs for what your server replies to googlebot.
Cloaked Redirects - .htaccess hacks
Google's John Mueller (JohnMu) has just made an excellent blog post about this. I'll refer you to him:
I urge you to read JohnMu's entire article [johnmu.com]. He's offering a lot of help here.
Some of the sneakiest hackers have used various kinds of DNS tricks. Over two years ago we discussed this rare but still possible problem in this thread [webmasterworld.com].
If your traffic totally dries up, you would hit the panic button pretty quickly - so these hackers have been more clever than that. With DNS tricks they might syphon off only 20% of your traffic. One thing you would see was a traffic drop with no corresponding drop in rankings.
There's been some good effort here on the part of the DNS servers to get more secure from this type of thing, but it's still worth mentioning as a potential. The moral is to check your DNS settings and fix any warnings you get. It might seem like a foregin language to you if you never waded into these waters before, but it's worth climbing the learning curve - especially if your traffic is evaporating. However, it's something that I wouldn't suspect until I ruled out all the rest of the hacks I listed above.
It might be an employee, too
Sorry to say, it's not always an external hacker. Sometimes a person your trusted with server access gets greedy and places parasite links to earn some csh on the side. We've had such reports here, and it even happened at Google a few years back.
Don't get crazy about this possibility, but if you do find junk on your server and there's no real sign of an external hack - then consider who you might have given server access to. This is one solid reason always to changes passwords (strong ones) when anyone leaves the company, or when your contract is over with anyone who had access. Even great companies sometimes hire a bad apple.