shri - 6:33 am on Dec 6, 2010 (gmt 0)

>> Our best effort at bringing it in house worked out to about $25,000 a year for reliable internet.

Unless prices are over inflated in the US, you're probably overthinking it.

We serve over 1M visitors a month over several sites from our office.

My connectivity is

2 x 10Mbps VDSL connections with 8IP each ( US$ 800 / month )
1 x 100 Mbps 1 IP ( US$ 100 each per month )

We spend an additional $150-200 on Amazon for S3 storage, static css / images / banners are all delivered via Cloudfront.

Equipment in the office

4 x Dual Quad Xeons (Dell 1950s) with varying disks..
1 x 2850 Dual Quad Xeon with quad NIC for the front end web server.
1 x Dual Xeon with 10TB raid for backups which are then further backed up to S3 storage

Power - dealt with by having UPS which gracefully powerdown the servers when hell breaks loose.

Have had about 30 mins downtime in a year ...

Brett - I do think it is totally worth having the setup in your office, as long as you have a tested backup plan. The control and freedom to expand, tinker and customize is worth it. We've moved from a LAMP stack to a F(reeBSD)N(ginx)Mysql(5.5)P(hp-fpm) stack and LOVE IT .. not many hosts if any would support this in a managed configuration.