BuiltWith

[lucy24]

IP: various in 18, but may come from elsewhere as well
UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.0; +http://builtwith.com/biup) Chrome/74.0.3729.131 Safari/537.36

The page in the UA suggests if you don’t want your site indexed [sic] by them,

User-agent: BuiltWith
Disallow: /

I do not perfectly see what good this would do, since I have never had a robots.txt request from this User-Agent (or for that matter from some other UA at the same IP, immediately prior to the page request--a rare but attested behavior from some robots). Maybe what they meant to say was

BrowserMatch BuiltWith bad_agent

Previous discussion takes us all the way back to 2011 [webmasterworld.com] and the ever-alert Pfui.

[phranque]

if you don’t want your site indexed [sic] by them

unlike a search engine, there is no relevant or useful information or context to index for the builtwith application or dataset without a crawl, so it certainly follows that a robots exclusion is effectively an indexing exclusion as well.

[tangor]

Curious minds ask: "But what does BuiltWith do?"

Checked logs ... not seen in most recent five years...

[not2easy]

BuiltWith offers a database so that people can look up what platform a given site is built with. Why? No idea.
I do not think they are very accurate.

[tangor]

Thanks!

[lucy24]

I do not think they are very accurate.

You mean they profess to tell you what CMS it uses? Or they look for headers that say “made with DreamWeaver v. 0.8”?

I looked up my personal site, since that was--inexplicably--the only one they crawled, and learned that
--site uses {name} analytics
--site is on {name} server type
--site is hosted by {name}
--DNS is provided by {name}
--site has an IPv6 address
--site defaults to SSL

So far, so good. And also, more mysteriously, “The website contains code that allows the page to support IPhone / Mobile Content.” Did they learn this by looking at the CSS and seeing it’s responsive? I hope so, because I can’t think of anything else it might mean.

Shrug.

They must have learned this on last week’s crawl, since nothing new showed up in logs just now. (Since it’s my personal site, I had only to glance at the timestamp on the log file, showing no recent activity.)