Forum Moderators: open

Message Too Old, No Replies

GarlikCrawler

         

keyplyr

8:27 pm on Oct 29, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




UA: GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
Protocol: HTTP/1.1
Robots.txt: Yes
Host: experian.com
185.26.92.0 - 185.26.93.255
185.26.92.0/23

Web security

Archived thread: [webmasterworld.com...]

lucy24

6:47 pm on Nov 16, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



On some days, they've used a different UA on robots.txt requests, though they say this is a mistake that will shortly be remedied:

185.26.92.4 - - [13/Nov/2016:10:01:13 -0800] "GET /robots.txt HTTP/1.1" 200 1337 "-" "Python-urllib/2.6"
185.26.92.4 - - [13/Nov/2016:10:01:13 -0800] "GET /fun/clients.html HTTP/1.1" 200 3852 "-" "GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)"

Host is different from the 2011 thread because they've only just been aquired by Experian. For the same reason, their Contact page may not work yet, though fortunately the address in the UA string does. I suspect the Experian business is closely linked with expanding to the US, where originally they were strictly a UK service.

Disclaimer: I am always strongly biased in favor of entities that answer email. But security services that ask for robots.txt are a bonus; many consider themselves exempt.