| 3:27 pm on Apr 26, 2012 (gmt 0)|
"blank user-agent" [google.com]
| 4:49 pm on Apr 26, 2012 (gmt 0)|
I apply an additional header test to these types of fetches. I've seen the satellite providers occasionally using blank UA and referrer when fetching/caching images. The other header test catches things like you posted 99.99% of the time and lets the legit cache fetches through.
| 5:24 pm on Apr 26, 2012 (gmt 0)|
Do you want to block search spiders, or just the ones that don't provide a moniker?
| 10:38 pm on Apr 26, 2012 (gmt 0)|
| 4:40 pm on May 9, 2012 (gmt 0)|
| 5:49 pm on May 9, 2012 (gmt 0)|
It doesn't matter what the reason the user has, if you block users because they do not enable cookies you're doing yourself a disservice. About 10% of users do not enable cookies.
| 8:01 pm on May 9, 2012 (gmt 0)|
Kendo - cookies are no guarantee. The technique may work with "real" bots but a LOT of scrapers maintain cookies - which is especially useful for dealing with IP-hoppers. :)
There is also a UK law now that says that web sites must offer an option to not accept cookies. This is very badly thought out (eg no mention of how US etc sites can be trapped) but it's bureaucrats so you can't really expect much logical thought on the matter. Nevertheless, a few web sites in the UK will implement either an option or dump cookies completely where they are of little use.