Welcome to WebmasterWorld Guest from 54.146.194.42

Forum Moderators: martinibuster

Message Too Old, No Replies

Stopping Yahoo Pipes bot hammering our feed

Daeth throes?

     
3:28 pm on Feb 15, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


I used to be grateful for Y! dispersing our daily feed, but not anymore.

How do you stop the compulsive Yahoo pipes bot hammering a feed?

It hits it every minute or two despite the feed only being updated once a day. I know it doesn't eat BW, but it does keep our poor server's CPU needlessly busy.

When we blocked Yahoo Pipes on UA, it continues unabated sucking up thousands of 403s all day long. Insanely inefficient: No wonder Y! folded.

Anybody know how to get Y! to turn the Pipes feed bot off?

They get our feed via Slurp anyway. (And Slurp is hardly relaxed).
9:58 pm on Feb 15, 2012 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3091
votes: 2


Block their IP(s) in a firewall?
3:26 pm on Feb 16, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member sgt_kickaxe is a WebmasterWorld Top Contributor of All Time 5+ Year Member

joined:Apr 14, 2010
posts:3169
votes: 0


If your feed is processed through feedburner there is an option to block yahoo pipes. It's in the "index" service next to "should search engines be allowed to index your feed" option. Of course if you use feedburner you want to re-route your existing feed so that it is only accessible through feedburner (feedburner feedsmith plugin for wordpress if that's what you use as cms).

Another option that requires some server resources is to use your robots.txt file. Not as efficient and open to being ignored but...

User-agent: Yahoo Pipes
Disallow: /

It may not be an actual Yahoo Pipes account that is hammering your server, it could be a widget built using the pipes technology. Something like searchmonkey etc. in that case add this to your htaccess file (and modify for searchMonkey version)...

SetEnvIfNoCase User-Agent "Yahoo! SearchMonkey 1.0" noMonkey
<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=noMonkey
</Limit>
4:33 pm on Feb 17, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


Thank you dstiles, I'm just blocking via htaccess on the UA pipes at the moment. I scan my access logs daily, and I'm tired of seeing all those pipes 403s! It's just so rude, and inefficient of Yahoo. Good riddance to them.

Sgt_Kickaxe:
Thank you for your kind suggestions, I'll see which will apply best in my case, and your notes will help others.

Funnily enough, I never even considered using robots.txt and it certainly seems to be the Y! pipes bot itself, so I'll have a try, and see if it obeys robots.txt (some hope!)

Nonetheless, I'll give it the chance: Just one!

We've long since blocked Feedburner, because I never liked their combination of feeds and email delivery.

Thanks again!