Complete .htaccess to block spammers and evil robots

help for blocking evil robots and spammers

[karlo]

Can someone please share a complete .htaccess file which will help my site increase it's search engine ranking and block all evil robots and email catchers/gathers?

[abbeyvet]

You are looking for a magic formula - there isn't one.

What you have in your .htaccess file is dependent on how your site is set-up and structured and what you want to do, which will be something different for just about every site.

There is no defintive list of all bad robots either, it changes all the time and anyway one man's bad robot is another man's friend.

Search Google for lists of bad bots, you will find plenty of them, but they are really just reference lists. You need to watch your stats and take it from there really.

[Dijkgraaf]

I doubt that there is anything you can do in .htaccess that would increase your search engine ranking, and if there were such a thing it would be a well guarded secret :-)

There is no magic bullet against evil robots and e-mail harvesters, however do a search for bot trap script in your favourite search engine and you will find various solutions of how to detect some bad bots and then deny them.

[Wizcrafts]

Karlo;
I think you will find that this classic WebmasterWorld thread, spanning 3 years and in 3 separate parts, will provide the answer to the second part of your question:

"A close to perfect .htaccess ban list"

Part 1: [webmasterworld.com...]
Part 2: [webmasterworld.com...]
Part 3: [webmasterworld.com...]

The list and techniques became more refined as time went by and we all experimented with variations. I developed my list as a contributor on that thread and still use it to this day.

My only opinion about your first question, about helping your SE ranking, is to be sure to use 301 permanent rewrites or redirects for aliased, moved or renamed files, and 410 for GONE files (HTTP 1.1 only - otherwise serve 404 Not Found). There are several threads in this Forum (92) pertaining to these server codes and their use in .htaccess.

Wiz