Welcome to WebmasterWorld Guest from 54.145.209.34

Forum Moderators: Ocean10000 & incrediBILL

Nominet Privacy Policy

Sniffer At Large

   
1:50 pm on Oct 2, 2012 (gmt 0)

5+ Year Member



This was visiting my .co.uk domains today. It hits each site 2-3 times, first with HEAD / then GET /robots.txt.

No referrer, UA is "http://www.nominet.org.uk/privacypolicy".

From the aforementioned page:

So we can monitor usage of .uk domains, Nominet is running a short trial to collect data on whether .uk domain names resolve, where they are hosted, whether they are used for email and/or whether a web site is in place.


I'm not sure how they determine whether robots.txt allows or disallows...
2:55 pm on Oct 2, 2012 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



There are numerous domain checking bots, not sure why this one would be handled any different than the rest.
8:53 pm on Oct 2, 2012 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



I noticed it last week, too. It arrived on an unexpected IP, blocked itself, and is now blocked by me.
6:31 am on Oct 3, 2012 (gmt 0)



Any IP ranges?
7:50 am on Oct 3, 2012 (gmt 0)

5+ Year Member



213.248.192.0/19, Nominet.org. I was visited by 213.248.202.17, 71, 72.

It's a pukka 'crawler', but it's not making many friends by hitting the root before asking for robots.txt, or despite robots.txt.

Seems to be doing just like the blurb says. Given they're the UK registry controller, I think they could do better.
8:28 am on Oct 3, 2012 (gmt 0)



Thanks iamzippy :)
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month