Facebot

from Facebook

[Pfui]

Hi, all. This just in --

66.220.159.115
Facebot/1.0

robots.txt? NO

IP resolves to: rx115.tfbnw.net
Range: 66.220.144.0 - 66.220.159.255
CIDR: 66.220.144.0/20

Single GET to bare html only. Also not clustered with Fb regulars:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
facebookplatform/1.0 (+http://developers.facebook.com)

No clue if related to this same-name bot (last updated 2013-04-18):

[sourceforge.net...]

[keyplyr]

What did happen to it[Google Web Preview]? I see it periodically in logs, but I'm ### if I can find how to invoke it

I don't think it's being used in organic search any longer. Seems to have disappeared since Bing started doing it - LOL

You can "invoke" it in GWT > Crawl > Fetch as Google > Fetch and Render

[Samizdata]

What did happen to it?

Apologies for the rhetorical question.

It was discontinued in April 2013, ostensibly due to "very low usage".

Nothing to do with webmaster resistance or copyright issues, it seems.

As for the subject at hand, we are officially informed that:

Facebot will respect your robots.txt settings

You may recall that Google Web Preview did no such thing.

It blithely pulled images and other assets from restricted directories.

Facebot apparently won't be doing that.

...

[Samizdata]

As it happens, Facebook has never stolen any traffic from me. - Samizdata

Ohhh yes they have.

I used to read your posts with interest and respect your opinion.

...

[keyplyr]

LOL, sorry I'm not a Facebook hater... I get too much revenue producing traffic from them.

[Samizdata]

A webmaster who assumes that all sites operate in the same way as theirs would be unwise.

One who asserts that they know more about my traffic than I do would be plain foolish.

I am not aware of any "Facebook haters" in this forum.

After approximately 300 hits on robots.txt in two days Facebot appears to have taken the day off.

Perhaps someone at the company is trying to fix it.

...

[lucy24]

You can "invoke" it in GWT > Crawl > Fetch as Google > Fetch and Render

There's a crucial difference between this new feature and the search-based Google Preview: "Fetch and Render" shows what the googlebot sees, which in turn is constrained by robots.txt permissions. Preview never followed robots.txt-- that is, it didn't disobey, it simply never asked-- because it saw itself as the human user's agent.

:: detour to check something ::

More Bing Preview at this point than Google-- which definitely doesn't correspond to overall search frequency!-- but the most recent Google Preview I see is just a few days ago. That's 66.249.93.72; earlier ones used the expected 64.233 range. I think everything in wmt uses the Preview UA as well.

It really does depend, not just on your site but on overall pages. If someone recommends an ebook on social media it doesn't do me any good, because the thing's PD anyway and they don't continue to the rest of the site. But I recently noticed a FB visit to one recently added page that's definitely a "Tell all your friends" kind of thing. So they're welcome to it :)

[keyplyr]

A webmaster who assumes that all sites operate in the same way as theirs would be unwise... After approximately 300 hits on robots.txt in two days Facebot appears to have taken the day off. - Samizdata

Fully agree. Facebot hit my robots.txt once all week. Sent me over 2k in traffic.

This critter is not going to send you traffic... One who asserts that they know more about my traffic than I do would be plain foolish. - Samizdata

Yup, you said it :)

[Samizdata]

Traffic from Facebook is sent by people linking to external sites (I get plenty of it).

Facebot, like the late unlamented Google Web Preview, sends nothing.

Hope this helps.

...

[keyplyr]

Well if you're going to get semantic, by your interpretation it could be said that traffic from Google or Bing is sent by people clicking on links... Google or Bing send nothing.

Regardless, so far Facebot has requested robots.txt, obeyed it and the sky has not fallen.

Hope this helps.

...

[dstiles]

Not sure if this is actually the reason for the new bot but in a threatpost posting today suggests it may be...

"A collection of privacy and consumer groups from the United States and Europe has asked the Federal Trade Commission to force Facebook to suspend a recently installed program that mines information on sites that users’ visit around the Web in order to serve them interest-based ads...

“Facebook already installs cookies and pixel tags on users’ computers to track browsing activity on Facebook[.]com and Facebook apps. If Facebook is permitted to expand its data collection practices, those cookies and pixel tags will also track users’ browsing activity on any website that includes a few lines of Facebook code."

http[://]threatpost[.]com/consumer-groups-urge-ftc-to-halt-facebook-data-collection-program/107478

(remove [] from link)