Should I be worried?

Many erroneous requests from Russia/Ukraine

[Rawrishly]

My site's error log shows lots of requests for wordpressy and the like urls, all coming from different IP addresses. They were all at the same time, but I don't think a single IP address was responsible for more than one erroneous request. All the IPs seem to be from Russia/Ukraine/Austria area. Our site is all handwritten, no cms, but should I still be worried about something like this showing up in our logs? There were 66 of these requests spread over an hour or so. Is it likely that this will come back in greater force? What can I do to prevent an attack like this? I'm very new to all of this, and just recently started getting into modifying our sites .htaccess file. Any advice you can give me would be greatly appreciated.
Example:

[Wed Jun 10 22:40:39 2015] [error] [client 178.16.95.14] File does not exist: /data1/www/example_com/blog, referer: http://example.com/blog
[Wed Jun 10 22:40:56 2015] [error] [client 178.120.100.74] File does not exist: /data1/www/example_com/wordpress, referer: http://example.com/wordpress
[Wed Jun 10 22:40:59 2015] [error] [client 46.164.213.2] File does not exist: /data1/www/example_com/wp, referer: http://example.com/wp
[Wed Jun 10 22:41:31 2015] [error] [client 37.212.106.118] File does not exist: /data1/www/example_com/news, referer: http://example.com/news
[Wed Jun 10 22:41:34 2015] [error] [client 217.150.62.25] File does not exist: /data1/www/example_com/site, referer: http://example.com/site
[Wed Jun 10 22:41:40 2015] [error] [client 83.149.35.85] File does not exist: /data1/www/example_com/blogs, referer: http://example.com/blogs
[Wed Jun 10 22:41:42 2015] [error] [client 213.33.245.37] File does not exist: /data1/www/example_com/home, referer: http://example.com/home
[Wed Jun 10 22:41:58 2015] [error] [client 91.232.157.40] File does not exist: /data1/www/example_com/cms, referer: http://example.com/cms
[Wed Jun 10 22:42:01 2015] [error] [client 178.207.44.66] File does not exist: /data1/www/example_com/main, referer: http://example.com/main
[Wed Jun 10 22:42:02 2015] [error] [client 95.71.119.33] File does not exist: /data1/www/example_com/articles, referer: http://example.com/articles
[Wed Jun 10 22:42:17 2015] [error] [client 195.66.65.70] File does not exist: /data1/www/example_com/blog1, referer: http://example.com/blog1

[wilderness]

There are so many threads here where Russian/Ukraine traffic is discussed.
Each webmaster must determine what is beneficial or detrimental to their own website (s).

Only you are able to realize what traffic is beneficial to your site (s).
1) do you even have a 'data1' directory?
2) do you utilize WordPress or WP?
3) Is traffic from this portion of the world beneficial?

To answer your question!
There's no harm in 404's (not found), However the absence of a 403 (denied) is likely to prompt repeated requests.

These days and given the numbers of WP sites, no matter what attempts (access restrictions) we make as webmasters the WP requests continue.

[not2easy]

I understand from your question that the site is hand coded html, not Worpress. My guess from the error log entries is that someone has programmed a botnet to search for vulnerable files. The changing IP addresses all working on the same agenda points to the botnet blame. You can get to understand botnets better in this recent article: [webmasterworld.com...]

To echo wilderness, there is no harm in 404 errors. I would take a look and see how your canonical redirect is set up because for some reason, the error log shows requests for URLs in your root directory without www. are being redirected to a higher level above root where data1 would be your username under many hosting setups. I am referring to the requested URLs for "/data1/www/example_com/" that claim to be referred from "http://example.com/" - it is best to find out whether this is just the way that your error logs are presented or whether any actual redirect is taking place. It is not that rare for error logs to include path info but example.com is not example_com on any host I've seen (yet).

I would take a look at your raw access logs for the corresponding time to see if you can't get more information about what is happening. Faked referrers are common, but how are they getting only "not found" errors for non-public files? The error log does not show the error number 404, so it may be another error being reported, that should be visible in your raw access logs, where it can also show you whether the referrer is faked.

[Rawrishly]

Wilderness, the data1 directory is correct, and we do not utilize WordPress. I cannot just block traffic from Russia/Ukraine, as we do get legitimate traffic from that area as well.
It seems that both of you feel I should just let it go, as there's no real harm in 404 errors. If that is the case, I will simply not worry about it unless there is a good reason I should.
Thank you both for your responses.

[not2easy]

That is not what I suggested at all. I do not see a "404" but IF that is what the error is, then it is not terribly important.

How and why it is shown as it is in your logs is more of a concern. My suggestion was to verify the traffic's behavior by viewing your raw access logs.

[lucy24]

The specific requests don't matter. What you need to pull out of it is information. Look at the IPs making the requests, and find out what they are. In rare happy occasions they will turn out to be servers; more often they're infected computers from human ISPs. (Does anyone know why computers from eastern Europe and select other places like Brazil are so inordinately infection-prone? I sure don't.)

Another thing you may choose to do is make rules based on the request. Sure, a request for wp-admin will eventually lead to a 404 since you do not, in fact, have this directory. But the server only finds this out by looking up the file. (It has to do this on every single request, because who knows, you might have added or removed something.) You could say

RewriteRule wp - [F]

meaning slam the door on the suckers-- or, for that matter

RewriteRule wp - [R=404,L]

(yes, that's a legitimate flag) which means "send this response right away and don't even bother looking". Some robots go away faster when they get a 404 response; if they get a 403 they think OK, what doesn't he want me to see? And then there's

RewriteRule wp http://127.0.0.1 [R=301,L]

and similar contemplate-your-navel redirects like sending them back to their originating IP. At this point it's a judgement call. If you don't feel like assessing the psychology of each individual robot, just do whatever is most pleasing to your own psychology and/or to your server's workload.

I do not see a "404"

Apache error logs don't include the numerical response code. "Request denied by server configuration" will be a 403; "File not found" will generally be a 404 unless it was an internal request that will be overrridden by some other response-- for example, a 403 followed by a futile search for an expected custom 403 document.

[Rawrishly]

They are 404 errors.
The IPs seem to be infected computers mostly.
So, returning a 404 with an htaccess rule is technically more efficient since the server doesn't bother searching for the file? And, you think it'd be a better idea to 404 rather than 403 because bots are sometimes intrigued by a forbidden page? Haha does redirecting to localhost actually cause the bots any grief or do you just get enjoyment from it?

[lucy24]

or do you just get enjoyment from it

Heh, heh. It would be a pretty stupid robot to automatically follow a redirect without stopping to look what it is-- but then, many robots are that stupid. But yes, this is where we get into judgement-call territory. I've met some individual robots in the past that seemed to give up faster if they got some type of redirect. Obligatory caution: resist the temptation to redirect to some outside entity such as fbi.gov or kgb.su* or whatever seems appropriate. This is Not Nice and can blow up in your face.

If you know in advance that the request will end up getting a 404-- and it's a common request you get every day-- you may as well put the response in manually, so the server doesn't have to check. (That's for unwanted robots. For legitimate visitors like the Googlebot, this is where you return a 410 meaning "It used to exist but I've intentionally removed it." Ordinary stupid Ukrainian robots would just get confused and/or suspicious.)


* I made that up.

[Rawrishly]

Thanks Lucy24. :)

[dstiles]

Lucy...
> Does anyone know why computers from eastern Europe and select other places like Brazil are so inordinately infection-prone?

A lot of them may be illegal copies of MS OS's. If they are, MS does not update them so exploits are more likely to happen. Include China/Indonesia/etc in that list. Shame about Brazil because the country is at least attempting to build a newer/better internet structure. I wish they'd hurry up; the one we're stuck with now is rubbish! :(

I would also suggest a lower level of internet threat awareness in those countries, despite some of the leading anti-virus and threat investigators living in Eastern Europe. It only takes opening a single email to install a virus and surrender a computer to a botnet (nice to see another one taken down a few days ago!).

I also wonder if AV software is so well installed and/or updated in those locations.

[keyplyr]

there is no harm in 404 errors

Agreed, however I run a local script each day and pull all the 404s to manually take a look, just in case I've misspelled a file name or other typo's.

As far as requests for CMS type files (blog, forum, news, wordpress, admin, wp-, login, shell, etc) since I don't use any of these, I block them at the server level. They are always hack probes. When I see these blocked requests in my logs, I usually look up the IP address to see if they are coming from servers or ISPs or compromised biz accounts. If servers, I add the IP range to my block list. This is an easy way to get server farm ranges :)

If a botnet, I sometimes will gather the IPs used and block them for a week if they are causing a lot of trouble, but this usually proves futile since compromised IPs are plentiful and can be bought very cheaply so they often come back from fresh IPs.

Does anyone know why computers from eastern Europe and select other places like Brazil are so inordinately infection-prone?

How do accounts become compromised so that that they become a drone for bad actors? Through email? From browsers visiting infected web sites? Probably both, but one thing is common, the users connect through some type of ISP. So my guess would be these countries are high risk because the ISPs there are lax in their security.

Web sites probably get compromised through the same hits that probe for vulnerability, and so on and so on... This would be a security failure with the server host.

Telecommunication standards are probably set at least partially by the country they do business in, so the geo trends may play a role. Anyway, that's my guess.

[lucy24]

If a botnet, I sometimes will gather the IPs used and block them for a week

I've taken to setting an environmental variable* for IPs from assorted places in Eastern Europe, and also for certain forms of Yandex query that only seem to be used by botnets. Any request from these areas gets explicitly redirected (not rewritten) to a detour page-- at which point, a genuine human might choose to click a link and go on to their originally requested page, having proven their humanity. So far none ever has, I don't think, but it relieves me of worry about locking out a legitimate human** trying to research some arcane subject from an unexpected geographic region.


* Imaginatively called bad_russia because, hey, I have to call it something. "That's racist." --Cinema Sins.
** Unless the legitimate human is in China. In that case, tough.

[Rawrishly]

Interesting. Do you explain to the user that you redirected them due to their geographic location, or do you just throw up a blank page with a link to the url they wanted to go to?

[lucy24]

:: shuffling papers ::

I’m Awfully Sorry {and same in currently three other languages, two of which use non-Roman script}

You’ve accidentally replicated the behavior of an undesirable robot, so we have to take this brief detour.

Then there's a link to the page they requested, and sometimes one to Yandex search if that's what they came from. Seems more polite than saying "It appears to me that you are an undesirable robot" ;)

I have extremely obscure content, so I really do bend over backward to keep from locking out real humans. Some people have different priorities. Not long ago I locked out a German human who was-- or claimed to be-- using FF 3.6. I felt bad, but hey, if you're going to allow your browser to send a fake UA string (the bits didn't match up), that's your own lookout.

[Rawrishly]

Yeah, you've gotta walk a fine line between insulting your users and keeping bots out.

[keyplyr]

As I've said before, I think it a mistake to include links back into your site in a 403 file. I keep it simple. If they get blocked, they're supposed to be blocked.

Also, I've never had a botnet come from Yandex servers, however I've seen spoofed UAs containing Yandex references.

[lucy24]

I've never had a botnet come from Yandex servers

I don't mean they claim to be from Yandex. I mean robots-- that is, infected browsers-- with bogus yandsearch referers. For a while, the most common format was a query that made it look as if they had searched for my site by name, which is too implausible even to consider.

[Hoople]

To me if they don't use a lot of bandwidth and aren't trying anything evil they get a free pass. Once they cross the line I step up my blocking.

I like that 127 redirect. Good one Lucy :-) Reminds me of the old DNS error in relation to email "I don't talk to myself".