Welcome to WebmasterWorld Guest from 18.104.22.168
For someone starting a business from scratch, perhaps making websites, go where your costs are low, not many people are online yet, and there is little competition, but growth potential.
I wouldn't go to NYC to start an SEO firm from scratch, but perhaps if I was a decent NJ based SEO company looking to grow, it'd be a good move.
You may also want to find a state with no income tax; Florida, for example. Just note that housing prices have shot up in many areas in recent years in FL, although good deals can still be found.
Forget about New Jersey unless you've got money to throw around. It's expensive here. =/ But if you really want to be a Manhattan big shot without spending Manhattan money, look into Jersey City.