homepage Welcome to WebmasterWorld Guest from 54.161.236.92
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

    
How to get Yahoo feed bots to stay away
Angonasec




msg:4624236
 1:26 pm on Nov 18, 2013 (gmt 0)

We stopped our daily feed a year ago, after we discovered it was being auto-scraped and published on Blogspot without permission.

Most feed bots stopped calling when they realised the feed wasn't being updated, but Yahoo's feed bots continue to hammer even when blocked. Thousands of hits a day for a year now.

Any ideas how to get Y! to stop their bot at source?

I've not tried contacting them directly yet.

98.139.134.97 - - [17/Nov/2013] "GET /feed.xml HTTP/1.1" 403 304 "-" "YahooCacheSystem"
67.195.51.215 - - [17/Nov/2013] "GET /feed.xml HTTP/1.1" 403 304 "-" "YahooCacheSystem"

Such a waste of their resources and ours.

 

Angonasec




msg:4625335
 8:01 am on Nov 22, 2013 (gmt 0)

Not to worry, I finally got Yahoo!'s foolish bot to recognize a 410 and it appears to be having a lie-down.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved