homepage Welcome to WebmasterWorld Guest from 54.211.219.68
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

    
Stopping Yahoo Pipes bot hammering our feed
Daeth throes?
Angonasec

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4418067 posted 3:28 pm on Feb 15, 2012 (gmt 0)

I used to be grateful for Y! dispersing our daily feed, but not anymore.

How do you stop the compulsive Yahoo pipes bot hammering a feed?

It hits it every minute or two despite the feed only being updated once a day. I know it doesn't eat BW, but it does keep our poor server's CPU needlessly busy.

When we blocked Yahoo Pipes on UA, it continues unabated sucking up thousands of 403s all day long. Insanely inefficient: No wonder Y! folded.

Anybody know how to get Y! to turn the Pipes feed bot off?

They get our feed via Slurp anyway. (And Slurp is hardly relaxed).

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4418067 posted 9:58 pm on Feb 15, 2012 (gmt 0)

Block their IP(s) in a firewall?

Sgt_Kickaxe

WebmasterWorld Senior Member sgt_kickaxe us a WebmasterWorld Top Contributor of All Time



 
Msg#: 4418067 posted 3:26 pm on Feb 16, 2012 (gmt 0)

If your feed is processed through feedburner there is an option to block yahoo pipes. It's in the "index" service next to "should search engines be allowed to index your feed" option. Of course if you use feedburner you want to re-route your existing feed so that it is only accessible through feedburner (feedburner feedsmith plugin for wordpress if that's what you use as cms).

Another option that requires some server resources is to use your robots.txt file. Not as efficient and open to being ignored but...

User-agent: Yahoo Pipes
Disallow: /

It may not be an actual Yahoo Pipes account that is hammering your server, it could be a widget built using the pipes technology. Something like searchmonkey etc. in that case add this to your htaccess file (and modify for searchMonkey version)...

SetEnvIfNoCase User-Agent "Yahoo! SearchMonkey 1.0" noMonkey
<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=noMonkey
</Limit>

Angonasec

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4418067 posted 4:33 pm on Feb 17, 2012 (gmt 0)

Thank you dstiles, I'm just blocking via htaccess on the UA pipes at the moment. I scan my access logs daily, and I'm tired of seeing all those pipes 403s! It's just so rude, and inefficient of Yahoo. Good riddance to them.

Sgt_Kickaxe:
Thank you for your kind suggestions, I'll see which will apply best in my case, and your notes will help others.

Funnily enough, I never even considered using robots.txt and it certainly seems to be the Y! pipes bot itself, so I'll have a try, and see if it obeys robots.txt (some hope!)

Nonetheless, I'll give it the chance: Just one!

We've long since blocked Feedburner, because I never liked their combination of feeds and email delivery.

Thanks again!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved