homepage Welcome to WebmasterWorld Guest from 23.22.173.58
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Nominet Privacy Policy
Sniffer At Large
iamzippy

5+ Year Member



 
Msg#: 4502977 posted 1:50 pm on Oct 2, 2012 (gmt 0)

This was visiting my .co.uk domains today. It hits each site 2-3 times, first with HEAD / then GET /robots.txt.

No referrer, UA is "http://www.nominet.org.uk/privacypolicy".

From the aforementioned page:

So we can monitor usage of .uk domains, Nominet is running a short trial to collect data on whether .uk domain names resolve, where they are hosted, whether they are used for email and/or whether a web site is in place.


I'm not sure how they determine whether robots.txt allows or disallows...

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4502977 posted 2:55 pm on Oct 2, 2012 (gmt 0)

There are numerous domain checking bots, not sure why this one would be handled any different than the rest.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4502977 posted 8:53 pm on Oct 2, 2012 (gmt 0)

I noticed it last week, too. It arrived on an unexpected IP, blocked itself, and is now blocked by me.

MxAngel



 
Msg#: 4502977 posted 6:31 am on Oct 3, 2012 (gmt 0)

Any IP ranges?

iamzippy

5+ Year Member



 
Msg#: 4502977 posted 7:50 am on Oct 3, 2012 (gmt 0)

213.248.192.0/19, Nominet.org. I was visited by 213.248.202.17, 71, 72.

It's a pukka 'crawler', but it's not making many friends by hitting the root before asking for robots.txt, or despite robots.txt.

Seems to be doing just like the blurb says. Given they're the UK registry controller, I think they could do better.

MxAngel



 
Msg#: 4502977 posted 8:28 am on Oct 3, 2012 (gmt 0)

Thanks iamzippy :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved