Page is a not externally linkable
blend27 - 8:27 pm on Jan 20, 2013 (gmt 0)
Blend27 - I have the range 98.137.206.0 - 98.137.207.255 blocked as a "bot" with the note: "several bot-UA hits but no valid rDNS". This still pertains so any bot from that IP range remains blocked until yahoo gets its act together.
I have about 3 dozen IPs with that "NOT Firefox" appended to that standard Slurp UA from 98.137.20[6-7].nnn, they all point to Hnnn.hlfs.bf1.yahoo(.)com [bgp.he.net...] At the same time, and this started on the 20th of Dec/2012 there are hits from 72.30.198.nnn with the same user agent(as well as normal Slurp UA). They also point to Hnnn.hlfs.bf1.yahoo(.)com - [bgp.he.net...] .
I've been allowing 72.30.0.0/16 for ages.
98.137.20[6-7]... gets the boot for now. IPs from this rage do request the images from the pages they try to crawl(one dynamically generated image URI, logged and matched with a cookie that is generated from the original page request).