Welcome to WebmasterWorld Guest from 23.22.46.195

Forum Moderators: bill & werty

How to stop RSS addict apps?

   
5:58 pm on Jul 3, 2014 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



I've got an RSS feed that only needs updating maybe every six months, tops. Thing is, some readers insist on hitting the file every four hours, 24/7. And last weekend, one started hitting it approx. every 10 minutes:

00:28:52
00:39:21
00:49:52
01:00:21
01:10:52
01:21:21
01:31:52
01:42:22
01:52:52
02:03:21
02:13:52
02:24:23
02:34:52
02:45:22
02:55:52
03:06:22
03:16:53
03:27:22
03:37:52
03:48:21
03:58:52
04:09:22
04:19:52
04:30:22
04:40:52
04:51:22
05:01:52
05:12:22

That UA was:

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36

But most abusers are:

Apple-PubSub/65.23

Along the way I added limiting directives with some effect --

<ttl>
<skipHours>
<skipDays>

-- but too many completely ignore them. So I've taken to 403'ing the worst abusers by Host/IP. And the most abusive one, above, I finally dropped via iptables.

I realize complaining about too many RSS hits is atypical because everyone wants the hits and stats. But what started as a convenience for infrequent visitors now just wastes resources.

Thoughts?
6:30 pm on Jul 3, 2014 (gmt 0)



I wouldnt deny traffic through iptables because you never know what users you are blocking in the future.

Why not monitor the traffic generated by the RSS feed? If it is too low just stop serving the RSS feed and serve a 404. Or you could move the RSS feed to a low budget VPS and serving a 301 redirect on the current RSS feed.
7:03 pm on Jul 3, 2014 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



- I reserve iptables for cesspool server farm CIDRs and the worst-of-the-worst individual IPs. Hitting a very infrequently-updated file every 10 minutes ad nauseam qualifies as the latter vis-a-vis our small, private server.

- The traffic to those subscribers that use respectful apps is fine, and appreciated on both sides so there's no need to stop it... but for the abusers.

- Alas, axing or removing or moving the feed doesn't stop the abusive hits that are already going on. And 404s, or 301s, like 403s, only add to server processes and log bloat.

I guess I'm simply hoping for some way to throttle non-conforming apps via the RSS file. Got a magic element, a way to make them mind? Because even an htaccess/mod_rewrite rule limiting access to the file based on time of day doesn't prevent the endless, mindless, useless hits the rest of the time.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month