homepage Welcome to WebmasterWorld Guest from 54.242.231.109
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Nominet Privacy Policy
Sniffer At Large
iamzippy




msg:4502979
 1:50 pm on Oct 2, 2012 (gmt 0)

This was visiting my .co.uk domains today. It hits each site 2-3 times, first with HEAD / then GET /robots.txt.

No referrer, UA is "http://www.nominet.org.uk/privacypolicy".

From the aforementioned page:

So we can monitor usage of .uk domains, Nominet is running a short trial to collect data on whether .uk domain names resolve, where they are hosted, whether they are used for email and/or whether a web site is in place.


I'm not sure how they determine whether robots.txt allows or disallows...

 

wilderness




msg:4503011
 2:55 pm on Oct 2, 2012 (gmt 0)

There are numerous domain checking bots, not sure why this one would be handled any different than the rest.

dstiles




msg:4503177
 8:53 pm on Oct 2, 2012 (gmt 0)

I noticed it last week, too. It arrived on an unexpected IP, blocked itself, and is now blocked by me.

MxAngel




msg:4503301
 6:31 am on Oct 3, 2012 (gmt 0)

Any IP ranges?

iamzippy




msg:4503310
 7:50 am on Oct 3, 2012 (gmt 0)

213.248.192.0/19, Nominet.org. I was visited by 213.248.202.17, 71, 72.

It's a pukka 'crawler', but it's not making many friends by hitting the root before asking for robots.txt, or despite robots.txt.

Seems to be doing just like the blurb says. Given they're the UK registry controller, I think they could do better.

MxAngel




msg:4503347
 8:28 am on Oct 3, 2012 (gmt 0)

Thanks iamzippy :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved