FeedFetcher-Google running amok

[Pfui]

Over 1,000 hits in the last five hours, first to / and to /rss (a tiny, 28k file that's updated every few months, tops), and in the last hour, just to /. Normally, FeedFetcher asks for /rss so infrequently I don't even notice. But this -- wow!

The hammering first appeared on Feb. 25 for a few hours across three 'sessions', then stopped cold. But even that was nothing like what's going on as I type -- hits every few seconds to minutes across four or five Google Hosts. I've yet to temporarily 403 the UA because I'm not confident that'll stop the hits anyway. Anyone else seeing this? TIA for your thoughts.

UA (always): FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
Hosts (examples; all legit):
rate-limited-proxy-66-249-90-35.google.com
rate-limited-proxy-209-85-238-4.google.com

[lucy24]

Ugh. I've had them blocked forever ... or so I thought. After some poring over headers and htaccess to figure out what, exactly, I'm blocking, it turns out the version I see is a lookalike. (Isn't it fun when a robot identifies itself with some other robot that you've already blocked? Like Chinese robots claiming to be from Baidu.)

Mozilla/5.0 (compatible; Feedspot/1.0 (+https://www.feedspot.com/fs/fetcher; like FeedFetcher-Google)

Happily they all come from 52, so no further action needed. But if need be, the element “FeedFetcher” could likewise be blocked.

66.249.80.0/20 is an icky range anyway. For every somewhat-legitimate googloid function, there are probably three I don't need or want.

[tangor]

The older I get the less patience I have.

Easier to nuke than try to puzzle out where the humans might be.*

^{* If any!}

[SumGuy]

"Feedfetcher is how Google crawls RSS or Atom feeds for Google News and WebSub. Feedfetcher stores and periodically refreshes feeds that are requested by users of an app or service. Only podcast feeds get indexed in Google Search; however, if a feed doesn't follow the Atom or RSS specification, it may still be indexed. Here are some answers to the most commonly asked questions about how this user-controlled feed grabber works. "

I don't think I've ever seen feed fetcher. But then again I have no content remotely like what feed-fetcher is designed for. Do you?

[not2easy]

WP integrates RSS feeds so unless it is disabled or not configured to limit, it shares the contents via feeds. That would be useful for constantly changing data, easier than scraping anyway.

[Pfui]

- Not a WP site; just plain ol' hand-rolled html with a bunch of Perl and a smattering of JavaScript.

- By way of an update, the thousands of 403'd FeedFetcher-Google hits stopped as suddenly as they'd started -- until two days ago when WHAM! Back again from three more of G's servers, hitting every second in concert. Finally had my SysAdmin killfile the three via /iptables (I don't stray/play above /public_html) Here they are:

209.85.238.130
209.85.238.131
209.85.238.132

A.k.a. this and its ilk:

rate-limited-proxy-209-85-238-131.google.com
FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)

I guess I'll continue to update /rss.xml in case anybody else actually looks at it (which is rarely and has been for years). Thanks to all for comments.

[lucy24]

209.85.238.130
209.85.238.131
209.85.238.132

What an annoying combination. To cover the whole thing you'd need 209.85.238.128/29 (i.e. 128-135) but this being g### they may use other bits of the range for unrelated things.

:: detour to raw logs ::

Feh. Various parts of 209.85.138--including the abovenamed 130 and environs--are used for Site Verification. * But not uniquely, so if you do block the range they'll check from the relevant bits of 66.249, 72.14 or 74.125.


* /googleblahblah.html, not to be confused with the 404 checker (which for the life of me I can't find in logs, though they can't have discontinued it).