I'm currently hosting two websites, one is a Lasso based e-commerce site. My hosting provider recently disabled my site entirely b/c his server was getting overloaded and it was affecting his other clients on the same box.
He found that a specific IP was hitting our site, he said that it "appeared" to be a bot. He then blocked this IP and restored our site but said if this continued he'd have to disable it again. He also implicated that it was my fault for not having a robots.txt implemented to stop this from happening.
I'm still relatiely new at web designing and don't have much experience with the robots.txt file, however, I've read the tutorial and it seems relatively straight forward. Even if I would have had a robots.txt file implemented it seems that this could have still happened, isn't that correct?
So some questions that I'm not sure of how to answer came to mind:
Are there adverse affects to leaving your entire site open to site bots? I've read the responses about protecting user data, etc., but if i don't have that kind of sensitive information to protect, what's the problem? Wouldn't it be better to let the bots see as much as possible for search engines?
What types of files would you always want to exclude from bot searches?
What files are useless to have them look at?
If a bot is pounding my site is it my responsibility to resolve this problem, or my ISP's? or both?
Thanks for any advice.