Welcome to WebmasterWorld Guest from 54.158.65.139

Forum Moderators: martinibuster

How to get Yahoo feed bots to stay away

   
1:26 pm on Nov 18, 2013 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We stopped our daily feed a year ago, after we discovered it was being auto-scraped and published on Blogspot without permission.

Most feed bots stopped calling when they realised the feed wasn't being updated, but Yahoo's feed bots continue to hammer even when blocked. Thousands of hits a day for a year now.

Any ideas how to get Y! to stop their bot at source?

I've not tried contacting them directly yet.

98.139.134.97 - - [17/Nov/2013] "GET /feed.xml HTTP/1.1" 403 304 "-" "YahooCacheSystem"
67.195.51.215 - - [17/Nov/2013] "GET /feed.xml HTTP/1.1" 403 304 "-" "YahooCacheSystem"

Such a waste of their resources and ours.
8:01 am on Nov 22, 2013 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Not to worry, I finally got Yahoo!'s foolish bot to recognize a 410 and it appears to be having a lie-down.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month