| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|How to get Yahoo feed bots to stay away|
We stopped our daily feed a year ago, after we discovered it was being auto-scraped and published on Blogspot without permission.
Most feed bots stopped calling when they realised the feed wasn't being updated, but Yahoo's feed bots continue to hammer even when blocked. Thousands of hits a day for a year now.
Any ideas how to get Y! to stop their bot at source?
I've not tried contacting them directly yet.
18.104.22.168 - - [17/Nov/2013] "GET /feed.xml HTTP/1.1" 403 304 "-" "YahooCacheSystem"
22.214.171.124 - - [17/Nov/2013] "GET /feed.xml HTTP/1.1" 403 304 "-" "YahooCacheSystem"
Such a waste of their resources and ours.
Not to worry, I finally got Yahoo!'s foolish bot to recognize a 410 and it appears to be having a lie-down.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved