Forum Moderators: DixonJones
I recently (2 weeks ago) put a site up live on the net. I have been monitoring my stats and I have had 500 hits in 2 weeks from a user agent claiming to be Wget. Im worried that someone is using this to copy my site and penalise me in the SERPS.
Can anyone give me some advice as to what to do? I have blocked it in my HTACCESS file but is this a good idea or am I just being paranoid?
Any help would be great,
Thanks
You might want to post a "Terms of Use" page, including a specification that scraping the site is not an acceptable use. Then people can't complain when they get banned.
You might also want to look into installing a "bad bot" script, which can automatically ban IPs exhibiting suspicious behaviour.
Eliz.