.htaccess block user agent ("wordpress/?*")

[Christaras]

Hello this is my first post and i am a newbie so bare with me.

I found this site from google who kept always bringing it on top for every query i made, so ofc i will come here and post :)

I learned lots of different things by reading this forum and i am thankfull for that, but since i am a newbie i couldn't find exactly what i was looking for and here comes the post.

A friend game me access to his GoDaddy domain to see if i could do anything to help him with his constant attacks and what i saw there at the logs was crazy for me which i want to share with you and if you can help me with it will be godsend.

I took the time to remove some colums btw (ip/time/get/referer when list at normal state/request?)

normal activity

IP - - [time] "GET ... HTTP/1.1" 200 3475 "referer" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.40 Safari/537.31" 0 "-" "/var/chroot/home/ (request?)" 158764
IP - - [time] "GET ... HTTP/1.1" 206 3273256 "referer" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.40 Safari/537.31" 0 "-" "/var/chroot/home/ (request?)" 2286835
IP - - [time] "GET ... HTTP/1.1" 200 9576 "referer" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3)" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 248179
IP - - [time] "GET ... HTTP/1.1" 200 4667 "referer" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3)" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 283993

under attack

IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.5.1; http://demo1.vbc-usa.com" 1 "x-httpd-php" "/var/chroot/home/ (request?)" 153394
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/342; http://musi.iptime.org" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 119453
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.8.1; http://www.dessertbulletblog.com" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 177382
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.7.1; http://www.visioncosmo.com" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 172344
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.5.2; http://w.ichurchcom.kr" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 127499
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.3.2; http://blog.manyinsoft.com" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 105851
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.5.1; http://bdirect.co.kr" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 184413
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.3.2; http://jjujjuba.net" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 194365
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.4.2; http://5gram.com" 4 "x-httpd-php" "/var/chroot/home/ (request?)" 209249
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.3.1; http://blog.illumos.org" 1 "x-httpd-php" "/var/chroot/ (request?)" 141490
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.7.1; http://moonausosigi.net" 0 "x-httpd-php" "/var/chroot/ (request?)" 159146
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.3.2; http://www.primemarineinc.com" 1 "x-httpd-php" "/var/chroot/ (request?)" 172296
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.4.2; http://owall.co.kr" 0 "x-httpd-php" "/var/chroot/ (request?)" 129841
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.8.1; http://www.mformation.com" 1 "x-httpd-php" "/var/chroot/ (request?)" 185156
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.5.1; http://collective-museum.org" 6 "x-httpd-php" "/var/chroot/ (request?)" 186746
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.8.1; http://blog.lotte.co.kr" 0 "x-httpd-php" "/var/chroot/ (request?)" 164203
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.5.1; http://demo1.vbc-usa.com" 0 "x-httpd-php" "/var/chroot/ (request?)" 328982
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/320; http://musi.iptime.org" 1 "x-httpd-php" "/var/chroot/ (request?)" 240144
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.7.1; http://www.edushare.kr" 7 "x-httpd-php" "/var/chroot/ (request?)" 391106
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.8.1; http://www.dessertbulletblog.com" 0 "x-httpd-php" "/var/chroot/ (request?)" 286155
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.3.1; http://blog.illumos.org" 0 "x-httpd-php" "/var/chroot/ (request?)" 175576
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.5.2; http://w.ichurchcom.kr" 0 "x-httpd-php" "/var/chroot/ (request?)" 625533
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.4.2; http://5gram.com" 1 "x-httpd-php" "/var/chroot/ (request?)" 585685
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.3.2; http://blog.manyinsoft.com" 0 "x-httpd-php" "/var/chroot/ (request?)" 501184

I used apache log viewer to view this and none of the custom views were able to show it correct, except combined which was cutting User-Agent after; .

Extra info: this is just a small part from the log, the one that made during the 3 hours attack was 1GB in size and even notepad++ could not load it and i had to use Large Text File Viewer in order to view and post this as a raw log so you can see what logs after User-Agent.

I took the time to visit those sites that are listed as User-Agent and they are normal wp blogs and some are blanks (just another wp)

So my question, is there a way to block WordPress from visiting and any booter that will come up in the future? Thanks in advance :)

[lucy24]

Edit: Whoops! Didn't realize we were onto a new page.

is this correct now like this ?

That should work. Assuming your error document is

example.com/403.html

The RewriteRule should properly say "403\.html" with escaped literal period. But this isn't one of those situations where failure to escape . can lead to "unintended consequences" (Apache euphemism for widespread death, destruction and wholesale calamity).

500 is a little worrying because it tends to mean something is really wrong. An intentional server error is more likely to come through as a 503, such as a mod_security block or a "closed for repairs" message.

Are you using your own ErrorDocument name, or the default set by your host? If it's the host's default, then there is almost certainly a <Files> envelope that overrides any "Deny from..." directives. But if it's your individual pagename, make sure you've got a <Files> envelope of your own.

[Christaras]

Whoops! did the same omg, i was refreshing the page and i was seeing my self at the bottom and didnt notice that you had already anwsered me in 2nd page ^^

godaddy went crazy, they even posted an alert for that, so 500 was because of godaddy and allowed the attacks to pass through. rules were fine before, no reason to do block every1, but it lasted less than an hour and when godaddy was back to its working state i logged in and put the new rule inside and everything is back to normal.

But honestly im thinking of Deny all IPs from cloudflare that i have in logs over the past 2 days and remove all other rules to avoid apache bleeding as wilderness whould say ^^

[wilderness]

"avoid apache bleeding"

FWIW, "bleeding" applies to the attacks (WordPress requests), as opposed to anything caused bt Apache.

2nd FWIW, the very large list of IP's and/or deny froms that you initially had in place could be reduced drastically by denying the Entire Class A's.
EX:
deny from 58. 59. 60. 61. 116. 117. 118. 121. 122. 123. 124.
deny from 125. 202. 203. 210. 211. 218. 219. 220. 221.

[Christaras]

Thank you wilderness,
I will use that also since as it seems deny does better job than any other rule. And yeah i was wondering how to minimize all that deny, so you are as always one step ahead ^^

Btw, i believe what i said about bleeding was correct for many reasons, first there was a bleeding attack over godaddy.
To be exact it was OpenSSL and Heartbleed vulnerability [support.godaddy.com ] and godaddy wasnt the only1 [forbes.com ].
This attack caused godaddy to have a 502 server status and for some reasons, the rules that were in my apache htaccess to block wordpress agent could not work and let the attack pass through. so there we have the bleeding ^^ , i did managed btw to log to godaddy with cpanel at some point before he close his services complete and updated .htaccess with only deny all and allow my ip only. as soon as i did that our page status went from 500 to 403 for all even tho godaddy was still having problems and eventually godaddy locked down any access from all.
So it took godaddy 1 hour to have the services back up and running and then i updated the rules as lucy said, fixing everything.
Even tho this wasnt something caused by the rules, but something that had to do with godaddy services, apache was able to work with deny than apply the rules while under attack.

Anyway, i believe this topic should be closed now.
I got more than enough info on my first question and i also believe that i will find a better section to have you all advising me on how to work with my attacks ? ^^ since you are all so kind and helpful :)

[Christaras]

Lucy are you here ? ^^

Something strange happened again without reason and i dont know how to fix it. Again with no reason attacks bypassed rules with a 500 status instead of 403. i believe that was because apache went overloaded with the error reports ^^. rule for 403 didnt make it better because i didnt write it correct.
what i did after was what i did last time, block all with deny and allow only me to access page untill i can figure out from logs a way to fix it. but instead of server going live it gives me a 503 status and all others a 403 as it should.
and the error log from my wrong 403 rule says RewriteRule: cannot compile regular expression '^.errordocs\\403.html'
if i remove ^ from start, then it goes like this: unable to include "/${GD_ERROR_DOC}" in parsed file (Absolute Hosting Path)/.errordocs/403.html

I removed all rules and kept only deny allow and keeps giving me a 503 status.
I dont know what to do :(

[lucy24]

'^.errordocs\\403.html'

What on earth is this intended to do? \\ means a literal backslash-- which should never occur in a request anyway, so why bother to include it? Maybe a careless cut-and-paste from a local file that used \ instead of / for directory slashes?

A 503 doesn't occur naturally. Anything other than a 404 ("can't find it") or 500 ("you goofed") means there's a rule specifying this response. Yes, OK, mod_dir returns a 301 in directory-slash redirects. Work with me here.

Does your host have mod_security enabled? That's generally the only time I see a 503, unless I've put up a temporary "under construction" type thing myself.

if i remove ^ from start, then it goes like this: unable to include "/${GD_ERROR_DOC}" in parsed file (Absolute Hosting Path)/.errordocs/403.html

"Unable to include..." means you've got an SSI but the server either can't find or isn't allowed to access the file. The error only occurs if includes are enabled; otherwise the server doesn't even try and there's no error.

What's that leading . in errordocs? Maybe you'd better cite the line where this reference occurs.

Thought. Does your custom 403 page have SSI content? If so, you need to add the [NS] flag to any RewriteRule creating a lockout. Or, probably easier, put an [L] line that says something like

RewriteRule ^includes/ - [L]

along with your other hole-poking exemptions at the beginning of your RewriteRules. You will also need one for lockouts that come from mod_authzzzz. These are most easily done by putting a supplementary htaccess file in your /includes/ (or whatever it's called) directory, containing just one directive:

Order Deny,Allow
Allow from all

If you have bad robots asking for your included files by name-- which shouldn't happen, because they've no way of knowing the files exist-- you have to get more complicated. But most of the time this is all you need.

[Christaras]

.errordocs was there from godaddy ?
'^.errordocs\\403.html' was from logs after i added ^403.html in rules.
when i changed it to 403.html without ^ it was unable to include (in error log).
i changed the rule to ^.errordocs/403.\html which i saw you writing at another post and im waiting to see the results ;p

503 is coming from godaddy who finaly informed us that he is updating all services to secure heartbleed ssl.

Now that you said SSI i figured out the weird part that i saw earlier before server 503 status.
while i was testing a way to fix the 403.html error i faced i wall while i was updating .htaccess file. i was getting an access denied to write over the file for no reason, and (my bad) i did replace it from cpanel file manager instead direct ftp without backing up that file and see what it had inside because i thought it was mine. was it ? :S
Now the strange part is that i found another one .htaccess in a the folder .errordocs and had the SSI handlers inside, aka

AddType text/html .html
AddHandler default-handler .html
AddOutputFilter INCLUDES .html

so there is an SSI and cant access it ?
and the 403 page does have SSI content <!--#include virtual="/${GD_ERROR_DOC}" -->

Do i need to change my main .htaccess file and add SSI handlers as well ? :p

like this ?
#
Options +Includes <-- is this thing needed here or inside .errordocs ? because neither had it
#
AddType text/html .html
AddHandler default-handler .html
AddOutputFilter INCLUDES .html
#
RewriteEngine on
#
RewriteRule ^.errordocs/403\.html - [L] <-or leave it 403.html now that we have includes ?
# IF User Agent "contains WordPress"
RewriteCond %{HTTP_USER_AGENT} WordPress
RewriteRule .* - [F]
#Only allow Googlebot from specific IP's
RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteCond %{REMOTE_ADDR} !^66\.249\.(6[4-9]|7\d)\.
RewriteRule .* - [F]
#

edit: while i was writing ... 503 status changed and let me in, so godaddy finished with his updates :/

[lucy24]

Aside from mod_rewrite, almost everything in Apache is inherited.

If you're using SSIs throughout the site, then set the option in your top-level htaccess. If only one directory uses includes, put a separate htaccess file in that directory. The line about "AddOutputFilter" is only needed if your include files have something other than the .shtml extension. (They probably do, unless you used includes from day one. The alternative is to change all your extensions, either by rewrite or redirect, which is an awful bother.)

<!--#include virtual="/${GD_ERROR_DOC}" --> 

For this you'll have to ask someone at GoDaddy, because it sure sounds like something GoDaddy-specific. In fact it sounds recursive: "Include the error document in the error document". They're one of the biggest hosts around, so someone will know.

:: detour to look up ::

Drat. I don't care who's best, I'm just asking who's biggest. Oh well.

[Christaras]

I'll upload the file with the rule that makes apache unable to include and i'll ticket GoDaddy to figure it out. :P

I hope no more hiccup from GoDaddy's end comes because whenever they do they let the attacks go through.
Maybe i should redirect them to any1 that wants traffic ? ;p
kidding :)

THANK YOU LUCY

[lucy24]

Maybe i should redirect them to any1 that wants traffic ?

Some robots do go away faster if you send them out to contemplate their navels at 127.0.0.1. Redirecting them to their own IP is another approach that has been suggested, though I've never personally tried it. If it's an extremely stupid robot, and they're on shared hosting, this would be a speedy way to get them kicked off the server :)

Do not repeat NOT redirect to any real-life site such as google. Save that kind of thing for when you get a scam phone call and you ask them to call back at your other number-- and give them the number of the local FBI office.