>> Our best effort at bringing it in house worked out to about $25,000 a year for reliable internet.
Unless prices are over inflated in the US, you're probably overthinking it.
We serve over 1M visitors a month over several sites from our office.
My connectivity is
2 x 10Mbps VDSL connections with 8IP each ( US$ 800 / month )
1 x 100 Mbps 1 IP ( US$ 100 each per month )
We spend an additional $150-200 on Amazon for S3 storage, static css / images / banners are all delivered via Cloudfront.
Equipment in the office
4 x Dual Quad Xeons (Dell 1950s) with varying disks..
1 x 2850 Dual Quad Xeon with quad NIC for the front end web server.
1 x Dual Xeon with 10TB raid for backups which are then further backed up to S3 storage
Power - dealt with by having UPS which gracefully powerdown the servers when hell breaks loose.
Have had about 30 mins downtime in a year ...
Brett - I do think it is totally worth having the setup in your office, as long as you have a tested backup plan. The control and freedom to expand, tinker and customize is worth it. We've moved from a LAMP stack to a F(reeBSD)N(ginx)Mysql(5.5)P(hp-fpm) stack and LOVE IT .. not many hosts if any would support this in a managed configuration.