Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: martinibuster

Message Too Old, No Replies

How to get Yahoo feed bots to stay away

     
1:26 pm on Nov 18, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


We stopped our daily feed a year ago, after we discovered it was being auto-scraped and published on Blogspot without permission.

Most feed bots stopped calling when they realised the feed wasn't being updated, but Yahoo's feed bots continue to hammer even when blocked. Thousands of hits a day for a year now.

Any ideas how to get Y! to stop their bot at source?

I've not tried contacting them directly yet.

98.139.134.97 - - [17/Nov/2013] "GET /feed.xml HTTP/1.1" 403 304 "-" "YahooCacheSystem"
67.195.51.215 - - [17/Nov/2013] "GET /feed.xml HTTP/1.1" 403 304 "-" "YahooCacheSystem"

Such a waste of their resources and ours.
8:01 am on Nov 22, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 13, 2003
posts:693
votes: 0


Not to worry, I finally got Yahoo!'s foolish bot to recognize a 410 and it appears to be having a lie-down.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members