Welcome to WebmasterWorld Guest from 54.145.65.62

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Nominet Privacy Policy

Sniffer At Large

     
1:50 pm on Oct 2, 2012 (gmt 0)

Junior Member

5+ Year Member

joined:Nov 1, 2006
posts: 66
votes: 0


This was visiting my .co.uk domains today. It hits each site 2-3 times, first with HEAD / then GET /robots.txt.

No referrer, UA is "http://www.nominet.org.uk/privacypolicy".

From the aforementioned page:

So we can monitor usage of .uk domains, Nominet is running a short trial to collect data on whether .uk domain names resolve, where they are hosted, whether they are used for email and/or whether a web site is in place.


I'm not sure how they determine whether robots.txt allows or disallows...
2:55 pm on Oct 2, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


There are numerous domain checking bots, not sure why this one would be handled any different than the rest.
8:53 pm on Oct 2, 2012 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3121
votes: 3


I noticed it last week, too. It arrived on an unexpected IP, blocked itself, and is now blocked by me.
6:31 am on Oct 3, 2012 (gmt 0)

Junior Member

5+ Year Member

joined:Aug 18, 2010
posts: 49
votes: 0


Any IP ranges?
7:50 am on Oct 3, 2012 (gmt 0)

Junior Member

5+ Year Member

joined:Nov 1, 2006
posts: 66
votes: 0


213.248.192.0/19, Nominet.org. I was visited by 213.248.202.17, 71, 72.

It's a pukka 'crawler', but it's not making many friends by hitting the root before asking for robots.txt, or despite robots.txt.

Seems to be doing just like the blurb says. Given they're the UK registry controller, I think they could do better.
8:28 am on Oct 3, 2012 (gmt 0)

Junior Member

5+ Year Member

joined:Aug 18, 2010
posts: 49
votes: 0


Thanks iamzippy :)