joined:Nov 1, 2006
This was visiting my .co.uk domains today. It hits each site 2-3 times, first with HEAD / then GET /robots.txt.
No referrer, UA is "http://www.nominet.org.uk/privacypolicy".
From the aforementioned page:
So we can monitor usage of .uk domains, Nominet is running a short trial to collect data on whether .uk domain names resolve, where they are hosted, whether they are used for email and/or whether a web site is in place.
I'm not sure how they determine whether robots.txt allows or disallows...