GarlikCrawler

[aristotle]

185.26.92.*
/ 
 Http Code: 200 Date: May 05 13:18:59 Http Version: HTTP/1.1 Size in Bytes: 9098 
 Referer: - 
 Agent: GarlikCrawler/1.2 (http://garlik.com/, [e-mail])

Robots.txt -- yes

From a search results snippet
"Garlik.com provides services that help protect online consumers from identity theft and financial fraud"

Don't know why it needs to crawl my website

[lucy24]

Don't know why it needs to crawl my website

Me neither, and I've got even less reason--I'm http and clearly don’t collect personal information. But it appears to be robots.txt compliant so what the heck.

:: detour to refresh memory ::

Oh, right. They're one of those robots that originally just swooped in to grab files, and then they discovered the joys of respectability and started asking for robots.txt around October of last year. Never a full top-to-bottom spidering; it looks like they start with individual pages, and then by and by they come back and request pages that are linked from that original page.

[aristotle]

lucy -- My site is http, static html, and doesn't collect any personal information as well. Over the past few hours, this bot crawled all 22 pages of this small site. I don't remember seeing it before on any of my sites..

[keyplyr]

Don't know why it needs to crawl my website

@aristotle - did you read the information found on their website? http://garlik.com or email them at the address they provide in their UA string? [e-mail]
They would be the one who could best explain why they are scanning your pages.

Doesn't matter if you collect user data through forms or not, data is collected by your server in the execution of serving web pages. The bot is checking for security during this process.

As for your site being HTTP (instead of HTTPS) this may be one of the red flags bringing this bot back to your pages. If you better protected your visitors by using HTTPS, you might not see this bot so often.

Previous discussion: [webmasterworld.com...]

[aristotle]

As for your site being HTTP (instead of HTTPS) this may be one of the red flags bringing this bot back to your pages.

Actually the opposite might be true. If someone sets up a site for the purpose of identity theft or financial fraud, most likely they would make it https in order to look legitimate. So http could actually be a sign that a site is harmless.

[keyplyr]

Not my point. HTTPS is secure. Among other things, this bot is scanning for unsecured HTTP vulnerabilities.

There is absolutely no advantage to serving unsecured HTTP files to your visitors, and soon all unsecured HTTP pages will display more severe warnings so you'll likely lose your traffic altogether.

[aristotle]

Well I thought they were focusing on identity theft and financial fraud, not on finding unsecure http sites

[keyplyr]

If your site is unsecure then it is at risk for visitors. Bots like this (there are many) may label your site as a risk and warn users or even remove it from the sites it says are safe.

Wouldn't it just be easier to switch to HTTPS? Doesn't take much work: What Will Happen if I Don't Switch to HTTPS? [webmasterworld.com]