Ip address keeps visiting my Apache server

[born2run]

So this ip address [snip] visits my apache server every few minutes.

Obviously I blocked it and now it keeps on going. Is there any page to report this ip address or further blocking?

I'm using AWS instance with apache server in it. Thanks!

[edited by: phranque at 11:13 pm (utc) on Jul 18, 2017]
[edit reason] see forum Charter [/edit]

[not2easy]

How did you block it? I mean what are you using to block it with?

[born2run]

I am behind Cloudflare so I blocked the ip from their firewall settings.

[not2easy]

If the CIDR is blocked, they may still appear in your logs. They should be receiving 403 responses rather than 200. You can check that in the access logs.

[keyplyr]

As not2easy says, blocking does not stop the agent from attempts to access your files, it stops successful attempts.

These unsuccessful attempts still show in your raw server logs along with other communications.

[TorontoBoy]

Send a note to [the hosting company's] Security and complain about [the Contact]. I've not complained to them in the past, so I don't know. You might get lucky. I find complaining to ISPs is a 50-50% success rate. Some ISPs will not do anything. Of course ISPs do not like when you ban their ranges, so have some incentive to do something.

[edited by: phranque at 7:19 am (utc) on Jul 19, 2017]
[edit reason] see forum Charter [/edit]

[phranque]

I blocked the ip from their firewall settings

unsuccessful attempts still show in your raw server logs along with other communications

if it's blocked at the firewall, it shouldn't get through to the web server - blocked IP addresses wouldn't appear in the web server log.

[lucy24]

Send a note to [the ISP's] Security

I don't think it is an ISP. I looked it up before the thread was edited, and I've got it down as a server range. Good luck blocking those, unless the robot is doing something over-the-top destructive like, say, making requests every few nanoseconds rather than every few minutes.

born2run, I don't think you ever said (pre-edit) what the visitor is requesting. Just the root, or something more interesting and/or complicated?

[keyplyr]

I don't think it is an ISP. I looked it up before the thread was edited, and I've got it down as a server range.

Yes, the IP belongs to Datashack, a large server farm.

Datashack ranges are listed here: [webmasterworld.com...]

[born2run]

So here's the details, it's using different UAs like:

(compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
(compatible; YandexImages/3.0; +http://yandex.com/bots)

Visits like every one minute at a time. I guess it's a bot that's not stopping.

[keyplyr]

born2run - these are common User Agents [webmasterworld.com] coming from common Server Farm Ranges [webmasterworld.com]

The information is here in the forums. You need to do some reading. Most all your questions have already been discussed, some many many times. Use the Search Utility in the upper-right corner.

[TorontoBoy]

(compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
(compatible; YandexImages/3.0; +http://yandex.com/bots)

These are completely different bots. In my .htaccess here's how I ban them:

SetEnvIf User-Agent AhrefsBot keep_out
order allow,deny
allow from all
deny from env=keep_out

I allow Yandex to index my site. There are a couple of different Yandex bots. They are not too bad.

[keyplyr]

Both of those bots obey robots.txt

No need to block, just disallow.

[born2run]

My point is that from one IP address two useragents are being seen. So it's a fake bot I presume

[keyplyr]

Well that's certainly a possibility. The Yandexbot comes from verifiable Yandex ranges.

The AhrefsBot used to come from Choopa ranges but could have moved to Datashack.

It really helps to start keeping notes on these agents, what they do, where they come from. After a while it all starts to make sense.

Also, since CDNs use multiple file servers from various points around the world, the log reporting can be unreliable. Things get changed around so it isn't consistent.... but you probably have found that out by now.

[born2run]

Thanks! ok I have another issue. lots of visitors being "challenged" by Cloudflare coz of this reason:

False IE6 detection [Type C]

What's this about anyways?

[lucy24]

False IE6 detection [Type C]

Heh. My host has a mod_security option, and one pattern they block is a UA string involving IE6. Wouldn't be surprised if Cloudflare looks for similar patterns.

:: shuffling papers ::

I think it's this, based on searching logs for 418 response + MSIE 6:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

Idle query: When did anyone last detect a non-false IE6? I don't think I've seen one since 2014.

[born2run]

Ok great thanks all, I'll have to monitor traffic every few hours to check what's been going wrong. Thanks again!

[born2run]

Yeah I complained to the owner of that IP address. So let's see in a few days.

[TorontoBoy]

I usually file a complaint with the IP address' host provider, not the actual IP address owner. I find that I receive no reply from IP address owners. The host provider can put pressure on the owner to change or get kicked off their host.

[born2run]

Yep I emailed the host provider too. No reply so far, and the ip is still visiting my site :-(