Do search engines penalize if you directing pages not found back to the home page? We would like to use our home page as our custom 404 page. Our webhosting company says that they can't or will not do 301 redirects.
It depends on whether the 404 page 1) is really a 404 page (i.e. returns a HTTP response code of 404, along with text which may e.g. be the same as on your home page), or 2) the custom '404 page' returns result code 200 (OK).
In the first case search engines won't index your error page's text (which is what you want), in the second case they will see your home page content duplicated at a number of other URLs on the site, and probably slap a negative weighting for duplicate content on your home page.