Forum Moderators: phranque
My site is on an Apache server. CPanel shows my bandwidth is getting leeched by mysterious spiders crawling my site. In addition I'm getting spam mails with virus attachement to them. In CPanel I have seen certain IP addresses that have spidered like 600 of my pages at a time, and this is killing my allowed bandwidth. My site used up 5 gigs in a matter of days. So I want to know how to go about stopping the spiders. Has it something to do with .htaccess? I keep seeing that referenced. Is there a file already made that I can use to stop the spiders and mail harvesters. Please help me, this is costing me money in excess bandwidth charges. Very disparaging I must say. If someone kind enough would post instructions or where I should look to put protections up it would be so cool and I could spend the money on beer rather than pay my host excessive bandwidth charges.
here are some threads/links to get you started, some heavy stuff in there.
Robots.txt Tutorial [searchengineworld.com]
Apache Tutorial: .htaccess files [httpd.apache.org]
A Close to perfect .htaccess ban list [webmasterworld.com]
