Was the domain name EVER registered to anyone else?
Is the site in a bad neighborhood (are there lots of bad sites hosted on the same server)?
does the site have the markings of a "legitimate" business? (Such as a physical street address, a phone number, etc.,)
and how does a site that only gets one or two visits a day get enough natural links to reach PR 4?
have you looked to see if anyone is building bad links to the site?