homepage Welcome to WebmasterWorld Guest from 54.227.215.139
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
katbot spider user agent and IP inforation
wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4653638 posted 12:07 pm on Mar 13, 2014 (gmt 0)

Anybody have any info? (I've broken the UA link).

The domain is registered to the NY Times, however the sub-domain results in 404.

The bot name (at least on a google) offers an entirely different content focus.

The RR-TWC hits were relevant and included all supporting files.

72.43.122.zzz - - [11/Mar/2014:20:22:23 -0600] "GET /MYFolder/MySub/MyPage.html HTTP/1.1" 200 12617 "http://example/com/MyOtherFolder/MyOtherPage.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; LEN2)"

184.73.62.101 - - [11/Mar/2014:20:39:31 -0600] "GET /SameFolder/SameSub/SamePage.html HTTP/1.1" 403 577 "-" "KatBot <katbot.nytlabs .com>"

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4653638 posted 10:51 pm on Mar 13, 2014 (gmt 0)

BTW, I'm aware this is from an Amazon IP.

Frank_Rizzo

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4653638 posted 7:10 pm on Jun 30, 2014 (gmt 0)

I got visited by it today.

It only accessed one page - a forum page, which recently linked to a NYT article. So I guess it's just an in house backlink checker.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved