Block Hetzner German ip?

[born2run]

So I'm getting lots of hits from this IP 5.9.32.***

I ran some searches on google and found it's listed as an abusive IP etc. Should I block this network? It's a huge hosting service it seems however.

[edited by: keyplyr at 5:27 am (utc) on Mar 7, 2018]
[edit reason] depersonalized IP address [/edit]

[keyplyr]

Hetzner is indeed a refuge for malicious agents *however* it is also home to many potentially beneficial agents, so you may need to do the research and allow some UAs through. This is true for any UA or IP range you block. Do not just start blocking without doing the research.

Hetzner has hundreds of IP ranges. They can be found all over the Server Farm IP Ranges [webmasterworld.com] thread.

[born2run]

From Hetzner comes a UA which is hitting my site quite often: Companybook-Crawler

I've noticed from search on WebmasterWorld keyplyr you suggested to block it. Any advice now? Thanks!

[born2run]

Thanks keyplyr i did some search on this site and blocked 2 UA so far based on stuff written here:

BUbiNG (+http://law.di.unimi.it/BUbiNG.html#wc)
CompanyBook Hetzner

Keyplyr you said earlier:
"I block all (known) University CS ranges. Let those future Sergey's test their projects on someone else's files."

How do I find all known University CS automated user agents? Is there any list? Thanks!

[not2easy]

See, the problem is that what I may regard as a UA to block may be just the UA you want to see crawling your site. My list is not your list. Each webmaster needs to research their access logs and know what traffic they want and what they do not want. Yes, it's time consuming to get started. If you just don't have the time, maybe you need an employee. All they could do is produce reports for you to make decisions on and there's software/scripts for reports. Again, what is convenient for one may not help another person; that's why there are those forums about Server Farms and UAs, to make the work co-operative.

[keyplyr]

From Hetzner comes a UA which is hitting my site quite often: Companybook-Crawler

I've noticed from search on WebmasterWorld keyplyr you suggested to block it. Any advice now? Thanks!

@born2run - I did not suggest to block it.

Understand... User Agents & IP ranges listed in these forums is for documentation only. No recommendation to block or allow is implied.

Related: [webmasterworld.com...]

[TravisDGarrett]

It might not be a good idea, but I ended blocking IP ranges from Hetzner, OVH, Online.net, also IP which are from China and Russia. I know this is not good idea, and certainly I am blocking legitimate traffic, but there are too many abuses from these IP ranges :-(

[keyplyr]

On one site I have, I'm blocking over 2k ranges, however I allow about 300 UAs in those block rules.

There are a dozen ISPs in Hetzner ranges, also at OVH. That means if you are blocking without prejudice, you are blocking thousands and thousands of potential website visitors who may click ads or buy products.

It pays to do the research.

[born2run]

Yes keyplyr is right. I've blocked useragents from searching this website for now. Let's see the effect for a few days. I'll be checking the list of UAs every couple of days or even daily! These bots or atleast some of them are creating havoc!

[TravisDGarrett]

There are a dozen ISPs in Hetzner ranges, also at OVH. That means if you are blocking without prejudice, you are blocking thousands and thousands of potential website visitors who may click ads or buy products.

It pays to do the research.

I know, but trust me, I've been working on this for now 10+ years, I even wrote PHP scripts, to try to detect unnatural behaviors, to block IP selectively, with a whole system of "temporary" blocking (because of course, offenders are often changing IPs), but "in my case", it was a lost cause. So I took the radical decision to block OVH, etc... I am certainly loosing some money, but in the other hand, I am not ending with my sites copied all over the web, at least way less, I am saving on bandwidth, and my pages are loading faster for real visitors.

Again this morning, when I woke up, there were 100.000 requests from 1.000+ IP from OVH ! (because I still keep an eye of what is being blocked). All those trying to download pages, or images, with user agent simulating real browser, and with also referrer URL (pages which have no links to my site).

At a time, I was reporting this to OVH 's abuse team, but this was totally useless.

[born2run]

What's ovh btw?

[TravisDGarrett]

What's ovh btw?

It's the largest host in Europe, and the third largest in the World*

* this can be subjective, I think it's based on the number of physical servers they host, but anyhow it's pretty big player.

[wilderness]

"From Hetzner comes a UA which is hitting my site quite often: Companybook-Crawler "

born2,
the simplest solution is to add word Crawl +(no case flag) to your denied UA's.
Besides stopping this one, you'll stop others (it's been a commonly used name by bots for more than 20-years).

[wilderness]

born2,
I've the entire 5 (Class A) denied for years.
Somebody will be along shortly an advise you that there are some slight detriments (that good guys exist there) in denying the '5', however it suits my purpose.

Only you may decide what is detrimental or beneficial.

[keyplyr]

the simplest solution is to add word Crawl +(no case flag) to your denied UA's.

That's one way to do it.

Besides stopping this one, you'll stop others (it's been a commonly used name by bots for more than 20-years).

That's the problem. Without extensive research, using a catch-all UA attribute block will also block potentially beneficial agents.


Again, if you start blocking, you need to manually examine your access logs constantly and research just who is getting block by those block rules.

[wilderness]

born2,
were I to include the following notation in after each and every word of my replies, it would not be enough justification for keyplr to miss noting exceptions (for that one (and rare) visitor that is actually beneficial to your website (s), as opposed to actually assisting you.

You must explore each one with your own raw logs and determine for yourself whether using each word is detrimental or beneficial.

60-70 of all website traffic is useless non-beneficial visitors and/or bots and may simply be denied. Course, if your looking for that one bot that is going to make you a billionaire, than please by all means comply with the rarest of exceptions.

[keyplyr]

@wilderness - Times change. Techniques successful years ago are no longer prudent. You don't publish Adsense or use other on-site marketing resources AFAIK, while born2run does.

Blocking with a broad brush does not work for today's environment, it needs to be more surgical than that. Using catch-all UA attributes will block too many potentially beneficial agents. What those are is specific to each site owners interests.

[lucy24]

I've the entire 5 (Class A) denied for years.

Obligatory caution: Wilderness has the rare advantage of running sites that have absolutely no benefit or advantage to humans outside North America, so he can categorically deny any and all non-ARIN addresses. That does make things a heck of a lot easier.

No two people have the same rules. (I, for example, admit almost everyone under the sun--including ones that would make, say, keyplyr recoil in horror--provided they only ask nicely ... and provided they are not from China.)

[wilderness]

"Obligatory caution: Wilderness has the rare advantage of running sites that have absolutely no benefit or advantage to humans outside North America, so he can categorically deny any and all non-ARIN addresses. That does make things a heck of a lot easier. "

lucy,
My sites provide a GIF help with an alternative email for contact. Widget users across the world (of which my escapades (lack of a better name for non-www-archives) are known may contact me and I'll poke a hole for access.
(if they wish the content, there are rarely other sources available.)

[SumGuy]

I block all Hetzner IP's as I encounter / discover them and add them permanently to my server's 403 list. Same with OVH and Linode.

[keyplyr]

Most all hosting companies & server farm ranges have been identified in the Server Farm IP Ranges forum [webmasterworld.com]

[wilderness]

SumGuy, I do similar, however it's never a one-size-fits-all and each webmaster must determine what is beneficial or detrimental to their own site (s).

Many thanks for your input.