homepage Welcome to WebmasterWorld Guest from 54.167.182.201
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Code, Content, and Presentation / RSS, ATOM, and Related Technologies
Forum Library, Charter, Moderators: bill & werty

RSS, ATOM, and Related Technologies Forum

    
How to stop RSS addict apps?
Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4684916 posted 5:58 pm on Jul 3, 2014 (gmt 0)

I've got an RSS feed that only needs updating maybe every six months, tops. Thing is, some readers insist on hitting the file every four hours, 24/7. And last weekend, one started hitting it approx. every 10 minutes:

00:28:52
00:39:21
00:49:52
01:00:21
01:10:52
01:21:21
01:31:52
01:42:22
01:52:52
02:03:21
02:13:52
02:24:23
02:34:52
02:45:22
02:55:52
03:06:22
03:16:53
03:27:22
03:37:52
03:48:21
03:58:52
04:09:22
04:19:52
04:30:22
04:40:52
04:51:22
05:01:52
05:12:22

That UA was:

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36

But most abusers are:

Apple-PubSub/65.23

Along the way I added limiting directives with some effect --

<ttl>
<skipHours>
<skipDays>

-- but too many completely ignore them. So I've taken to 403'ing the worst abusers by Host/IP. And the most abusive one, above, I finally dropped via iptables.

I realize complaining about too many RSS hits is atypical because everyone wants the hits and stats. But what started as a convenience for infrequent visitors now just wastes resources.

Thoughts?

 

bhukkel



 
Msg#: 4684916 posted 6:30 pm on Jul 3, 2014 (gmt 0)

I wouldnt deny traffic through iptables because you never know what users you are blocking in the future.

Why not monitor the traffic generated by the RSS feed? If it is too low just stop serving the RSS feed and serve a 404. Or you could move the RSS feed to a low budget VPS and serving a 301 redirect on the current RSS feed.

Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4684916 posted 7:03 pm on Jul 3, 2014 (gmt 0)

- I reserve iptables for cesspool server farm CIDRs and the worst-of-the-worst individual IPs. Hitting a very infrequently-updated file every 10 minutes ad nauseam qualifies as the latter vis-a-vis our small, private server.

- The traffic to those subscribers that use respectful apps is fine, and appreciated on both sides so there's no need to stop it... but for the abusers.

- Alas, axing or removing or moving the feed doesn't stop the abusive hits that are already going on. And 404s, or 301s, like 403s, only add to server processes and log bloat.

I guess I'm simply hoping for some way to throttle non-conforming apps via the RSS file. Got a magic element, a way to make them mind? Because even an htaccess/mod_rewrite rule limiting access to the file based on time of day doesn't prevent the endless, mindless, useless hits the rest of the time.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / RSS, ATOM, and Related Technologies
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved