Stopping Yahoo Pipes bot hammering our feed

Daeth throes?

[Angonasec]

I used to be grateful for Y! dispersing our daily feed, but not anymore.

How do you stop the compulsive Yahoo pipes bot hammering a feed?

It hits it every minute or two despite the feed only being updated once a day. I know it doesn't eat BW, but it does keep our poor server's CPU needlessly busy.

When we blocked Yahoo Pipes on UA, it continues unabated sucking up thousands of 403s all day long. Insanely inefficient: No wonder Y! folded.

Anybody know how to get Y! to turn the Pipes feed bot off?

They get our feed via Slurp anyway. (And Slurp is hardly relaxed).

[dstiles]

Block their IP(s) in a firewall?

[Sgt_Kickaxe]

If your feed is processed through feedburner there is an option to block yahoo pipes. It's in the "index" service next to "should search engines be allowed to index your feed" option. Of course if you use feedburner you want to re-route your existing feed so that it is only accessible through feedburner (feedburner feedsmith plugin for wordpress if that's what you use as cms).

Another option that requires some server resources is to use your robots.txt file. Not as efficient and open to being ignored but...

User-agent: Yahoo Pipes
Disallow: /

It may not be an actual Yahoo Pipes account that is hammering your server, it could be a widget built using the pipes technology. Something like searchmonkey etc. in that case add this to your htaccess file (and modify for searchMonkey version)...

SetEnvIfNoCase User-Agent "Yahoo! SearchMonkey 1.0" noMonkey
<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=noMonkey
</Limit>

[Angonasec]

Thank you dstiles, I'm just blocking via htaccess on the UA pipes at the moment. I scan my access logs daily, and I'm tired of seeing all those pipes 403s! It's just so rude, and inefficient of Yahoo. Good riddance to them.

Sgt_Kickaxe:
Thank you for your kind suggestions, I'll see which will apply best in my case, and your notes will help others.

Funnily enough, I never even considered using robots.txt and it certainly seems to be the Y! pipes bot itself, so I'll have a try, and see if it obeys robots.txt (some hope!)

Nonetheless, I'll give it the chance: Just one!

We've long since blocked Feedburner, because I never liked their combination of feeds and email delivery.

Thanks again!