I would say vBulletin is fairly extensible. First, there are lots of third-party plugins already available. There's a big developer community, too.
The only caution on modifying vB is that it may make upgrading more difficult. Still, if you can do your mods as plugins that will simplify things.