Facebook cortex

[lucy24]

IP: {facebook ranges}
UA: cortex/1.0

I’ve seen this a couple of times in recent weeks (March 2019), always mixed in with a longer visit involving known FB UAs. I don’t get a lot of facebook, so it may be a little older, though probably not a lot.

[tangor]

Thanks for the eagle eye, lucy24 ... made me look. My time frame is very close to yours (Feb 2019, last week of the month).

[wilderness]

solitary request in Feb 2019 and was immediately following a duplicate request for an image where the Blank UA was denied.

[lucy24]

where the Blank UA was denied

Yes, I've had to poke a hole in my no-UA ban purely for Facebook’s benefit. This annoys me.

[wilderness]

lucy,
Kudos to you.
I recently added that IP's can into my denies. Their simply no benefit to my site. Course, everyone is not me.

[lucy24]

Frankly, there is no earthly reason why facebook can’t choose to be robots.txt compliant, as the twitterbot is. But it does occasionally result in added human visitors, so they’re exempt.

Now, here’s a scary thought. (Whoops! This is April Fool's Day and it ought to be Halloween for what I am about to say.) Suppose the Googlebot suddenly decided not to bother about robots.txt compliance. What would webmasters do?

[tangor]

That would be expelling bricks in distress ... (chose to keep the language clean). Otherwise, it would be a very messy experience!

[iamlost]

@lucy24: Suppose the Googlebot suddenly decided not to bother about robots.txt compliance. What would webmasters do?

1. run rDNS sequence as usual to confirm if legit.
2. if not legit block.
3. if legit allow except block as per ignored robots.txt directives.
Always hope for best behaviour but prepare for worst.