Best method to block unwanted visits via Cpanel

Site is routinely being overwhelmed with UAs(?) I don't care about

[millimi1]

I'm not an IT person, so please bear with me. That said, I know enough to follow along.

I'm a small scale webmaster (and I use that term lightly). I literally have one site – my church – that I host & manage their webpage as part of my giving. Its a WordPress site and I manage it through the Cpanel interface. A few weeks ago my provider reached out about high usage on my site that was tripping the CPU usage fault on their server. This has happened before and I generally block a few IPs and it goes away. This time however, they are asking me to upgrade my plan to provide more bandwidth, but I know I don't need it. This is a small church (<100 people) with few online users. We are located in southern California and have a handful of folks elsewhere in the US and also Northern Europe and Australia. The website is mainly in place to help drive traffic to the church (weddings, receptions, events, church services, etc.). I want the site to be crawled by sites (Google, Bing, and such) that will help improve web searches and promote the site within our area, but realize most of this traffic is not doing that. Looking at my visit stats I'm seeing hits outside the US, mainly from China & Germany (neither of which I care about), but also sites that are supposedly US-based IPs that tie to companies like Hetzner Online GmbH. I've blocked huge swaths of IPv4s. At the recommendation of my host, I also have the G7 firewall in my .htaccess and have a robots.txt file with crawl-delay and disallow requests for numerous paths in the site as well as certain bots. I also use the Wordfence Security plugin within WP. Wondering if there is a more efficient way within Cpanel to block this unwanted traffic. For instance, I don't see a way of saying "block all traffic from China" or XYZ company.

I appreciate any guidance and apologize if this topic is already covered.

[not2easy]

Hi millimi1 and welcome to WebmasterWorld [webmasterworld.com]

Bots are currently hitting hard across the world. Cloudflare can make it easy to block traffic from entire areas of the globe when that traffic is unwanted. They offer various levels of services starting with free service. It's free to look into and if you were to search for "Cloudflare" here on the site search you will likely find only happy users. Search is shown in the page header menu here.

[tangor]

Cloudflare is one option. Another is to visit the Spiders, Crawlers and Bots section here at WW (use the search box noted above)

A half hundred "rules" in .htaccess can kill SIGNIFICANT amounts of this unwanted traffic and keep server loads (and sever logs) under control.

There are nation/state level denies possible, something covered in fair detail in the Apache Forum here at WW if you have no concerns over possibly blocking "humans".

The described site sounds like a labor of love, not a commercial entity, so putting in personal time (and learning new things) might be of more value than relying on a third party where there's only so much can be done for free before having to step up into paid.

Bots are pests. Kind of like roaches you can't kill them all, but you can kill enough that you can have a fairly clean house. And, like roaches, you have to keep after it else the little buggers mutate and the battle is on once again.

[lucy24]

And now the bad news:

#1 There is little significant difference in bandwidth between a blocked robot and one that gets in. Either way, they make a request and you send a response. (A blocked human uses much less bandwidth, because they'll also request supporting files like images and stylesheets. The vast majority of malign robots don’t.)

#2 That same vast majority of malign robots don’t even look at robots.txt, let alone honor its directives. This is not to say robots.txt is useless: some unwanted robots do follow instructions, and the only thing better than a blocked request is one that is never made in the first place.

That being said, there is definitely something in the air. On my own small site, I've lately been getting a hundred-plus non-blocked robots daily, where the norm has been at most half a dozen or so. (Is there some new robot script going the rounds? I wouldn’t be surprised.)

For instance, I don't see a way of saying "block all traffic from China" or XYZ company.

A rough-and-dirty way of blocking an entire country, if you can’t keep track of all its IP ranges, is to block by Accept-Language header. Blocking from a particular company can be done, but only at the cost of going into lookups mode on all requests, which might end up creating just as much work for your server.

For specifics, wander over to the Apache subforum (or IIS if that's what you are on).

[millimi1]

Thanks all for the insights & info. I'll take a look at Cloudflare as well as the Apache subforum here at WW. Thank you for the warm welcome & assistance. And yes, this is definitely a labor of love. I don't make a dime on any of this and I pay for the domain & hosting myself not to mention the hours I spend with upkeep and adding weekly content (sermon audio, videos, and various social media shorts).

[tangor]

Wishing all luck in taking on the Bot Battle.

The first step is identifying WHAT to address---that is a whole adventure in itself! Please take advantage of the resources here at WW, some of us have been doing this for DECADES!