Nominet Privacy Policy

Sniffer At Large

[iamzippy]

This was visiting my .co.uk domains today. It hits each site 2-3 times, first with HEAD / then GET /robots.txt.

No referrer, UA is "http://www.nominet.org.uk/privacypolicy".

From the aforementioned page:

So we can monitor usage of .uk domains, Nominet is running a short trial to collect data on whether .uk domain names resolve, where they are hosted, whether they are used for email and/or whether a web site is in place.

I'm not sure how they determine whether robots.txt allows or disallows...

[wilderness]

There are numerous domain checking bots, not sure why this one would be handled any different than the rest.

[dstiles]

I noticed it last week, too. It arrived on an unexpected IP, blocked itself, and is now blocked by me.

[MxAngel]

Any IP ranges?

[iamzippy]

213.248.192.0/19, Nominet.org. I was visited by 213.248.202.17, 71, 72.

It's a pukka 'crawler', but it's not making many friends by hitting the root before asking for robots.txt, or despite robots.txt.

Seems to be doing just like the blurb says. Given they're the UK registry controller, I think they could do better.

[MxAngel]

Thanks iamzippy :)