how to get rid of robots

[lucy24]

Haha, just kidding, I'm not going to tell the botrunners everything I know. Though admittedly your average robot is too stupid to live, so it wouldn't matter how much information you handed them on a plate.

I've just wrapped up a project that started about a month and a half ago.

Old htaccess shared by 6 sites:
29.4k
500-odd lines, including
57 lines of bad user-agents
14 lines of bad referers

New shared htaccess:
4.5k
150 lines, including
16 lines of bad user-agents
1 (one) bad referer

Time will tell, but a quick look at access logs from shortly after The Great Change reveals at least one unwanted visitor 403'd, and at least one law-abiding robot admitted.

Tralala.

[keyplyr]

And here I thought The Great Change was Bruce to Caitlyn

Seriously, congratulations. I know that is a lot of work .

[blend27]

Mine is 2,093 bytes and 66 lines (half are comments).

Requests get rewritten based on bad UAs and referrers to use a self maintained(programmatically in a separate memory space from main application) MAP file that contains list of bad IPs.

Then a few lines in the main application to catch bad headers, unwanted Country specific visitors, SQL injections, etc... Those few lines control the MAP file and record all the site access info, JUST because I want to know EBBBERY-ZING.

And it is in real time too!

But then again, there is always someone smarter that me out there... and the best advice I've heard to this day was, wait for it.....waaaiiiit......:

Put some Windex on it!

[keyplyr]

My file currently is weighing in at 257k and Google says my pages are very fast. I only use 2 mods.

When I went responsive a couple years ago, I got rid of several scripts and moved it to RegEx.

But really, with SSD, the size of the htaccess file is not so much an issue any longer. It's more of just better management on the admin side.

And of course it depends on what traffic your site gets. Playing the Social Media game changes the dynamic considerably.

[lucy24]

Playing the Social Media game changes the dynamic considerably.

For me it was the endless whack-a-mole with new IP ranges. More about who you are, less about where you come from.

[keyplyr]

More about who you are, less about where you come from

"It isn't where you came from, it's where you're going that counts." - Ella Fitzgerald