.htaccess block user agent ("wordpress/?*")

[Christaras]

Hello this is my first post and i am a newbie so bare with me.

I found this site from google who kept always bringing it on top for every query i made, so ofc i will come here and post :)

I learned lots of different things by reading this forum and i am thankfull for that, but since i am a newbie i couldn't find exactly what i was looking for and here comes the post.

A friend game me access to his GoDaddy domain to see if i could do anything to help him with his constant attacks and what i saw there at the logs was crazy for me which i want to share with you and if you can help me with it will be godsend.

I took the time to remove some colums btw (ip/time/get/referer when list at normal state/request?)

normal activity

IP - - [time] "GET ... HTTP/1.1" 200 3475 "referer" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.40 Safari/537.31" 0 "-" "/var/chroot/home/ (request?)" 158764
IP - - [time] "GET ... HTTP/1.1" 206 3273256 "referer" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.40 Safari/537.31" 0 "-" "/var/chroot/home/ (request?)" 2286835
IP - - [time] "GET ... HTTP/1.1" 200 9576 "referer" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3)" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 248179
IP - - [time] "GET ... HTTP/1.1" 200 4667 "referer" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3)" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 283993

under attack

IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.5.1; http://demo1.vbc-usa.com" 1 "x-httpd-php" "/var/chroot/home/ (request?)" 153394
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/342; http://musi.iptime.org" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 119453
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.8.1; http://www.dessertbulletblog.com" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 177382
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.7.1; http://www.visioncosmo.com" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 172344
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.5.2; http://w.ichurchcom.kr" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 127499
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.3.2; http://blog.manyinsoft.com" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 105851
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.5.1; http://bdirect.co.kr" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 184413
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.3.2; http://jjujjuba.net" 0 "x-httpd-php" "/var/chroot/home/ (request?)" 194365
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.4.2; http://5gram.com" 4 "x-httpd-php" "/var/chroot/home/ (request?)" 209249
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.3.1; http://blog.illumos.org" 1 "x-httpd-php" "/var/chroot/ (request?)" 141490
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.7.1; http://moonausosigi.net" 0 "x-httpd-php" "/var/chroot/ (request?)" 159146
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.3.2; http://www.primemarineinc.com" 1 "x-httpd-php" "/var/chroot/ (request?)" 172296
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.4.2; http://owall.co.kr" 0 "x-httpd-php" "/var/chroot/ (request?)" 129841
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.8.1; http://www.mformation.com" 1 "x-httpd-php" "/var/chroot/ (request?)" 185156
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.5.1; http://collective-museum.org" 6 "x-httpd-php" "/var/chroot/ (request?)" 186746
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.8.1; http://blog.lotte.co.kr" 0 "x-httpd-php" "/var/chroot/ (request?)" 164203
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.5.1; http://demo1.vbc-usa.com" 0 "x-httpd-php" "/var/chroot/ (request?)" 328982
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/320; http://musi.iptime.org" 1 "x-httpd-php" "/var/chroot/ (request?)" 240144
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.7.1; http://www.edushare.kr" 7 "x-httpd-php" "/var/chroot/ (request?)" 391106
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.8.1; http://www.dessertbulletblog.com" 0 "x-httpd-php" "/var/chroot/ (request?)" 286155
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.3.1; http://blog.illumos.org" 0 "x-httpd-php" "/var/chroot/ (request?)" 175576
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.5.2; http://w.ichurchcom.kr" 0 "x-httpd-php" "/var/chroot/ (request?)" 625533
IP - - [time] "GET ... HTTP/1.1" 200 3152 "-" "WordPress/3.4.2; http://5gram.com" 1 "x-httpd-php" "/var/chroot/ (request?)" 585685
IP - - [time] "GET ... HTTP/1.1" 200 3153 "-" "WordPress/3.3.2; http://blog.manyinsoft.com" 0 "x-httpd-php" "/var/chroot/ (request?)" 501184

I used apache log viewer to view this and none of the custom views were able to show it correct, except combined which was cutting User-Agent after; .

Extra info: this is just a small part from the log, the one that made during the 3 hours attack was 1GB in size and even notepad++ could not load it and i had to use Large Text File Viewer in order to view and post this as a raw log so you can see what logs after User-Agent.

I took the time to visit those sites that are listed as User-Agent and they are normal wp blogs and some are blanks (just another wp)

So my question, is there a way to block WordPress from visiting and any booter that will come up in the future? Thanks in advance :)

[aristotle]

Just out of curiosity,
Did the site get knocked completely offline?

Did Godaddy try to fight off the attack?

Did it just last 3 hours?

Did you mean that there were other attacks before this one?

[wilderness]

WordPress, SMF and a few other blog formats, have a section within their panel that offers restrciting visitors. Unfortuately it's bulky to use and even bulkier to keep track of.

Either of these methods should suffice temporarily, and until you get a grasp on the actual cause.
They are NOT intended to be permanent and merely used to stop the bleeding.

Order Deny,Allow
# IF User Agent "contains WordPress"
SetEnvIf User-Agent WordPress keep_out
Deny from env=keep_out

OR

RewriteEngine on
# IF User Agent "contains WordPress"
RewriteCond %{HTTP_USER_AGENT} WordPress
RewriteRule .* - [F]

[Christaras]

@aristotle yes attack was 3 hours long and thus made the log file 1gb and lots of lines with the wordpress as user agent

godaddy is the only host that would allow our forums to be hosted there. others dropped us after the first month because they couldnt handle the attacks that being made everyday for an hour up to three :/


@wilderness thank you, i will use what you said and see what happens,
we are using smf so if you have any extra setting that could be put there are welcome :)

btw site is under cloudflare also and its useless :/

[wilderness]

Christaras,
The difference in attacks/crawls is similar, however different than those presented to a standard website.

About two years ago, I lessened the attacks on a friend's SMF site utilizing htaccess rather than the SMF software (which my friend was using previously).

I'd suggest you determine the audience of you SMF/WordPress site (based upon world-regions) and then begin restricting the visitors by Class A IP's.

[aristotle]

godaddy is the only host that would allow our forums to be hosted there. others dropped us after the first month because they couldnt handle the attacks that being made everyday for an hour up to three

That's too bad. I'm worried about a possible attack against one of my sites, and it's hosted at a small company. I sure hope I don't have to move it, especailly to GoDaddy.

[lucy24]

Tangentially: Can someone who speaks Apache explain what the last 4 log entries are? I realize you have a lot of leeway in setting log format, but I'm just not used to the extra stuff after the UA. Especially on shared hosting, where they try to keep things to a minimum. In particular, what's that last huge number?

[Christaras]

Hm, today they changed User-Agent to look like googlebot/2.1
to be more specific it says Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Isn't there anything i can do to stop them ? are they just gonna keep changing user agent every time to pass restrictions?

@wilderness btw how do i block this User-Agent?[OR] ^Googlebot ?

I will have to disable cloudflare in order to determine audience and i dont have access for that, ill have to tell owner to do it :/ so i can see the real IP range instead of cloudflare IPs

[lucy24]

Do you have any RewriteRules in place already? If not, putter around this subforum to look at the preliminaries.

A fake user-agent is easy when they're spoofing something that's well-known in its real form. The basic form looks like

:: shuffling papers ::

RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteCond %{REMOTE_ADDR} !^(66\.249|74\.125)\.
RewriteRule (^|\.html|/)$ - [F]

That's my own version. To reduce work for the server, I limit most rules to requests for pages; mine happen to use the .html extension.

I don't think I really need the "74.125" part; it's just habit. Happily google, unlike some search engines one could name, is very consistent in its IPs.

Edit:

^Googlebot 

NOOO! The opening anchor means "the element 'Googlebot' has to come at the very beginning of the UA string" -- which, of course, it never does. Unless that's a characteristic of your particular spoofers. In which case, by all means block them.

[Christaras]

I updated .htaccess with both User-Agent but it doesnt seem to work, page still unavailable.

thank you lucy24

edit:
i said something stupid, removed it :p

[wilderness]

btw how do i block this User-Agent?[OR] ^Googlebot ?

lucy provided a focused example.
Here's a wider range:

RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteCond %{REMOTE_ADDR} !^66\.249\.(6[4-9]|[7][0-9])\.
RewriteRule .* - [F]

I updated .htaccess with both User-Agent but it doesnt seem to work, page still unavailable.
I think smf is messing with it since

You likely have a syntax error in place.

1) Processing order is Apache root and/or htaccess
2) Followed by SMF restrictions.

Your SMF may have an htaccess in a sub-directory with restrictions (or not) which override the root htaccess. If so, than you'll need to correct these.

Note; whilst your making these changes and/or testing. It's a good practice to keep copies of the old htaccess files.

[Christaras]

Yes thank you wilderness,

I saw the file htaccess.txt inside forums folder which is a mess inside, i changed it to match the structure of apache one and now im seeing error logs growing fast in size and normal apache logs staying the same. I also made owner to disable cloudflare at the same time so im not sure if this is because of cloudflare or the .htaccess beeing effective.
will wait and see, but web page still says unavailabe :/

[Christaras]

Ok let me update you,
I was reading the error logs which are growing in size very fast btw, it changed from "user limit" when it had no restrictions in .htaccess to "client denied by server configuration" after updating .htaccess so thats something :)
Yet all the ip's inside are still from cloudflare which shouldnt since owner already disabled it. so what is going on now ? ^.^ my head, sorry guys and thanks again

[wilderness]

Who cares where the IP's came from (their not accurate anyway due to cloudflare).

"client denied by server configuration"

Your slamming the door in face of the WordPress requests, which is what you desired.
You've stopped the bleeding.

The next thing you need to do is inform your registered SMF users that a configuration change is being tested, and should they have trouble accessing the site, than they should contact the forum administrator. (You'll then make an exception for their IP Range [good luck with that due to cloudflare] after verifying that the user is NOT one of the pests).

I've no knowledge of CloudFlare and despite seeing a few requests here, this is an Apache Forum.

[Christaras]

Sorry i forgot to say that server is still unavailable and the attack seems to still be ongoing because of the error logs growing fast (there are 4 logs inside error logs folder all 4 updating at same time and are daily with sizes 1.44GB,1.08GB,2GB,1.05GB) and the apache logs which are 100 btw but only the last 4 are updating which are the daily also ofc are now staying at same size whitout any change since none can access site. ^^

all i did was add at the top of .htaccess
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} WordPress
RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteCond %{REMOTE_ADDR} !^66\.249\.(6[4-9]|[7][0-9])\.
RewriteRule .* - [F]

[wilderness]

all i did was add at the top of .htaccess

In reality, and when you say "server is still unavailable"!

What you have are 503's, where every visitor is being denied because, rather than modifying the htaccess correctly you've created a syntax error.

You'll need to start a new htaccess from scratch (SAVE COPIES OF THE OLD ONES).
1) Determine what lines from the old htaccess are necessary
2) add the additional lines in their proper place.
3) Test that site after every htaccess update to confirm the site is working.

FWIW, denying access to visitors does NOT stop the requested attacks, nor the showing of the same in the raw logs or error logs.
Rather, it prevents their access through the server.

[wilderness]

RewriteCond %{HTTP_USER_AGENT} WordPress
RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteCond %{REMOTE_ADDR} !^66\.249\.(6[4-9]|[7][0-9])\.
RewriteRule .* - [F]

This is NOT what I provided!
You'll never have a request where the UA contains both WordPress and Googlebot.

Don't confuse what you have with the following, which is two entirely different rules.:

RewriteEngine on
# IF User Agent "contains WordPress"
RewriteCond %{HTTP_USER_AGENT} WordPress
RewriteRule .* - [F]
#Only allow Googlebot from specific IP's
RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteCond %{REMOTE_ADDR} !^66\.249\.(6[4-9]|[7][0-9])\.
RewriteRule .* - [F]

[wilderness]

Furthermore, I'd suggest you restore the original htaccess until you do two things.

1) remove the CloudFlare (allowing the showing of actual IP's)
2) Get your new htaccess assemebled and functioning properly.

[Christaras]

Yes sorry, did a new clean file with only what you wrote inside now and removed the one from smf inside forum which had many rules inside and realy bad like the ones i did and now it seems that web is up and running ^^

THANK YOU VERY MUCH wilderness

[wilderness]

Now that you have a functioning htaccess that should be accomplishing the denials you desire?

Suggest you go back and review the original file (before you started) and determine what that file contains that is necessary.

Some lines are missing for canonical domain.
There may also be some SMF lines that are necessary for the site forum to function properly.

[Christaras]

Yes, you are as always correct, website is not stable atm, it goes up/down by the minute.
It's not because the missing file needed by smf because i get the same results from main page which isnt controlled by smf, im gonna leave it as it is for now and let the logs build a little more to further study them later.
Once again thank you very much.

[lucy24]

RewriteCond %{HTTP_USER_AGENT} WordPress
RewriteCond %{HTTP_USER_AGENT} Googlebot

By default, all RewriteConds are [AND] meaning that all conditions have to be met. Here you simply want two entirely separate rulesets, as discussed above.

When you attach [OR] to the end of a RewriteCond, it means "either this or the following condition has to be met". Or one of a long string of conditions, each with [OR] at the end. It's closely analogous to using a pipe | in a Regular Expression. In fact if you're looking at two of the same thing, like two different UAs or two different IPs, a pipe is better. Save the [OR] flag for when you've got two entirely different things, or for when the line becomes so long that it's hard to read.

Caution: If you rearrange conditions and accidentally end up with an [OR] flag at the end of the last RewriteCond, the one immediately before the RewriteRule, your server will explode.

[wilderness]

your server will explode

Not literally, of course ;)

It will definitely cause a 503 and take the server down.

[Christaras]

Hello, thank you guys again for all the help you gave me last night and sorry for the trouble i caused you.
Website is up and running stable btw, thanks to you.

My problem is solved for now and i just want to ask a few more questions ^.^ before closing this topic.

Latest apache log that goddady gives me is a 3hours long, (btw im not sure why goddady breaks them in 4 files and updates them all at same time, i just took the newest one) with all the lines inside this 533MB file being filed with the same repeatable attack which i cant determine where is coming from because all IP's are from cloudflare that is now disabled, filters off and dns directly to goddady.
You can view a screenshot with part of apache log here [s25.postimg.org ]
it has status 403 for the attacker so its working
my question for this, is there any way to block booters ?
im sure that this scale of an attack can only be made by booters. im i wrong ?

And last one, tho it shouldnt be here since its about smf but forgive me please since it has to do with smf htaccess.txt.
I uploaded the file here [pastebin.com ] it is the file that i now have renamed it in order for smf not to load it and overwrite apache rules and i want to ask if i added in this file the new rules correct, so i can enable it back again. i think its wrong ^.^

[wilderness]

Those two rules that I provided, as well as all of your deny froms should be placed prior to the pretty URL rules (it makes no since to use the server to rewrite lines for visitors that are going to be denied in the end).

I've no background with the Pretty URL's, perhaps lucy may help.

[lucy24]

# If mod_security is enabled, attempt to disable it.

WHAT? Sure, you might need to disable it if it's blocking some specific request that you don't want to block. But otherwise let it do its thing, so requests get intercepted in the server before ever reaching your own site directory.

Get rid of the <IfModule> envelopes. Either you've got the mod or you haven't. In the case of mod_rewrite, you've obviously got it or the site wouldn't exist. And besides, the closing </IfModule> tag is in the wrong place.

#
deny from 180.76.*.*
#
deny from 208.177.76.7
#
deny from 208.177.*.*

This notation is wrong, and I don't want to think about what it does to the server. IPv4 addresses can stop at any point, so simply
Deny from 180.76
Deny from 208.177

This is CIDR notation, not Regular Expressions. So if you said
Deny from 208.1
it would only block 208.1.x.y, not 208.17.x.y or 208.177.x.y or 5.208.1.x

#
RewriteCond %{HTTP_USER_AGENT} WordPress
#
RewriteRule .* - [F]
#

#
RewriteCond %{HTTP_USER_AGENT} Googlebot
#
RewriteCond %{REMOTE_ADDR} !^66\.249\.(6[4-9]|[7][0-9])\.
#
RewriteRule .* - [F]

Rules ending in [F] should go before all other RewriteRules except the one poking a hole for your custom 403 page. Otherwise you're wasting time rewriting or redirecting requests that will end up being locked out.

\.(6[4-9]|[7][0-9])\.

=
\.(6[4-9]|7\d)\.

I've no background with the Pretty URL's, perhaps lucy may help.

Keeping in mind that my gut reaction to any extensionless URL is "Go back in the server and put some clothes on!"

[Christaras]

Hello again.

Thank you both lucy24 and wilderness for your support.

I want to say something that kept me all day facepalming myself ^^ but if you please, pretend that it never happened.
The forum is not mine as i said before and its not smf, its MyBB :/ i still dont underdand why i said smf...

OK, back to topic, i made a cleaner htaccess.txt for MyBB just to add back the calendar rules and the rest which im not sure why are there or who put them, but just to make sure ;p
Forum works even with the file missing from its directory but i want to add it back only for the SEO_sUPPORT and DEFLATE.
So thanks to Lucy's share of knowledge i made this [pastebin.com ] , is it ok to put it back? inside MyBB forum folder and let it overwrite apache's .htaccess simple rules we made before ?

Btw, does the previous \.(6[4-9]|[7][0-9])\. soon to be changed to \.(6[4-9]|7\d)\. has anything to do with the error AH00124 ? because apache keeps reporting this since the simple 2 rules alone were added.

Once again thank you all for your help and support, this community is the best :)

[aristotle]

Lucy wrote:
A fake user-agent is easy when they're spoofing something that's well-known in its real form. The basic form looks like
:: shuffling papers ::
RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteCond %{REMOTE_ADDR} !^(66\.249|74\.125)\.
RewriteRule (^|\.html|/)$ - [F]

That's my own version. To reduce work for the server, I limit most rules to requests for pages; mine happen to use the .html extension.

Lucy -- May I ask if there's a way to limit a rule to a request for one specifc page? For example, in the last line of your above code, could you replace |\.html|with the URL of a specific page?

[lucy24]

in the last line of your above code, could you replace |\.html|with the URL of a specific page?

Yes, absolutely. The more tightly you constrain a rule, the better.

---

does the previous \.(6[4-9]|[7][0-9])\. soon to be changed to \.(6[4-9]|7\d)\. has anything to do with the error AH00124

AH what now? I have no idea what that is :(

:: detour to look up ::

Oh. I didn't know it had a name.

Request exceeded the limit of 10 internal redirects

That means there's a mis-configured rule that leads to an infinite loop. I can't see how it would ever happen in a rule ending in [F].

Wait, stop, rewind. Yes I do. This is why you MUST have a preliminary rule that says, without condition,

RewriteRule ^forbidden.html - [L]

substituting the name of your own custom 403 page. This goes at the very beginning of all your RewriteRules.

Here's why:

Unwanted visitor makes a request.
mod_rewrite runs though its list of rules, finds one saying this visitor is not allowed, issues a 403
Server says OK, that means I send out the 403 page instead, and makes an internal request for this page.
mod_rewrite reads this internal request, runs though its list of rules, finds one saying this visitor-- meaning the original visitor, not the server-- is not allowed, issues a 403
Server says OK, that means I send out the 403 page instead, and makes an internal request for this page.
mod_rewrite reads this request, runs though its list of rules, finds one saying this visitor is not allowed, issues a ...

... et cetera until the server cuts off at-- typically-- 10 iterations.

[Christaras]

lucy ^^
I guess this is the reason why server had a funny behavor and allowed for some reason WordPress with a 500 status this time instead of 403..
and yeah page is down again ^^ and now i Deny all so that i can view the logs and see what to do.
but if i add now that preliminary will this cause more trouble now that i have set it to access denied for all ?


edit:

is this correct now like this ?

RewriteEngine on
#
RewriteRule ^403.html - [L]
# IF User Agent "contains WordPress"
RewriteCond %{HTTP_USER_AGENT} WordPress
RewriteRule .* - [F]
#Only allow Googlebot from specific IP's
RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteCond %{REMOTE_ADDR} !^66\.249\.(6[4-9]|7\d)\.
RewriteRule .* - [F]
#