The one thing I would do is to make sure the page is loaded before exposing any objects/embeddings. And activate the video on a click event with some js instead of having it automatically play right after page load.
No we're not. We thought about it and then realized there is just too much involved and ended up using a third party video hosting service in addition to YouTube and Ustream. We have 10+ TB of data being served. Not something you'd want to do yourself unless you were a video hosting expert.
How many videos are we talking about? How large are the files? What's wrong with using a professional video hosting service?