homepage Welcome to WebmasterWorld Guest from 54.227.146.68
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 40 message thread spans 2 pages: < < 40 ( 1 [2]     
Facebot
from Facebook
Pfui




msg:4679860
 11:49 am on Jun 14, 2014 (gmt 0)

Hi, all. This just in --

66.220.159.115
Facebot/1.0

robots.txt? NO

IP resolves to: rx115.tfbnw.net
Range: 66.220.144.0 - 66.220.159.255
CIDR: 66.220.144.0/20

Single GET to bare html only. Also not clustered with Fb regulars:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
facebookplatform/1.0 (+http://developers.facebook.com)

No clue if related to this same-name bot (last updated 2013-04-18):

[sourceforge.net...]

 

keyplyr




msg:4681282
 7:30 pm on Jun 19, 2014 (gmt 0)


What did happen to it[Google Web Preview]? I see it periodically in logs, but I'm ### if I can find how to invoke it

I don't think it's being used in organic search any longer. Seems to have disappeared since Bing started doing it - LOL

You can "invoke" it in GWT > Crawl > Fetch as Google > Fetch and Render

Samizdata




msg:4681293
 7:42 pm on Jun 19, 2014 (gmt 0)

What did happen to it?

Apologies for the rhetorical question.

It was discontinued in April 2013, ostensibly due to "very low usage".

Nothing to do with webmaster resistance or copyright issues, it seems.

As for the subject at hand, we are officially informed that:

Facebot will respect your robots.txt settings

You may recall that Google Web Preview did no such thing.

It blithely pulled images and other assets from restricted directories.

Facebot apparently won't be doing that.

...

Samizdata




msg:4681295
 7:52 pm on Jun 19, 2014 (gmt 0)

As it happens, Facebook has never stolen any traffic from me. - Samizdata

Ohhh yes they have.

I used to read your posts with interest and respect your opinion.

...

keyplyr




msg:4681301
 8:55 pm on Jun 19, 2014 (gmt 0)

LOL, sorry I'm not a Facebook hater... I get too much revenue producing traffic from them.

Samizdata




msg:4681321
 10:23 pm on Jun 19, 2014 (gmt 0)

A webmaster who assumes that all sites operate in the same way as theirs would be unwise.

One who asserts that they know more about my traffic than I do would be plain foolish.

I am not aware of any "Facebook haters" in this forum.

After approximately 300 hits on robots.txt in two days Facebot appears to have taken the day off.

Perhaps someone at the company is trying to fix it.

...

lucy24




msg:4681339
 11:28 pm on Jun 19, 2014 (gmt 0)

You can "invoke" it in GWT > Crawl > Fetch as Google > Fetch and Render

There's a crucial difference between this new feature and the search-based Google Preview: "Fetch and Render" shows what the googlebot sees, which in turn is constrained by robots.txt permissions. Preview never followed robots.txt-- that is, it didn't disobey, it simply never asked-- because it saw itself as the human user's agent.

:: detour to check something ::

More Bing Preview at this point than Google-- which definitely doesn't correspond to overall search frequency!-- but the most recent Google Preview I see is just a few days ago. That's 66.249.93.72; earlier ones used the expected 64.233 range. I think everything in wmt uses the Preview UA as well.

It really does depend, not just on your site but on overall pages. If someone recommends an ebook on social media it doesn't do me any good, because the thing's PD anyway and they don't continue to the rest of the site. But I recently noticed a FB visit to one recently added page that's definitely a "Tell all your friends" kind of thing. So they're welcome to it :)

keyplyr




msg:4681427
 7:13 am on Jun 20, 2014 (gmt 0)


A webmaster who assumes that all sites operate in the same way as theirs would be unwise... After approximately 300 hits on robots.txt in two days Facebot appears to have taken the day off. - Samizdata

Fully agree. Facebot hit my robots.txt once all week. Sent me over 2k in traffic.

This critter is not going to send you traffic... One who asserts that they know more about my traffic than I do would be plain foolish. - Samizdata

Yup, you said it :)

Samizdata




msg:4681473
 11:27 am on Jun 20, 2014 (gmt 0)

Traffic from Facebook is sent by people linking to external sites (I get plenty of it).

Facebot, like the late unlamented Google Web Preview, sends nothing.

Hope this helps.

...

keyplyr




msg:4681593
 7:14 pm on Jun 20, 2014 (gmt 0)

Well if you're going to get semantic, by your interpretation it could be said that traffic from Google or Bing is sent by people clicking on links... Google or Bing send nothing.

Regardless, so far Facebot has requested robots.txt, obeyed it and the sky has not fallen.

Hope this helps.

...

dstiles




msg:4691458
 2:12 pm on Jul 29, 2014 (gmt 0)

Not sure if this is actually the reason for the new bot but in a threatpost posting today suggests it may be...

"A collection of privacy and consumer groups from the United States and Europe has asked the Federal Trade Commission to force Facebook to suspend a recently installed program that mines information on sites that users’ visit around the Web in order to serve them interest-based ads...

“Facebook already installs cookies and pixel tags on users’ computers to track browsing activity on Facebook[.]com and Facebook apps. If Facebook is permitted to expand its data collection practices, those cookies and pixel tags will also track users’ browsing activity on any website that includes a few lines of Facebook code."

http[://]threatpost[.]com/consumer-groups-urge-ftc-to-halt-facebook-data-collection-program/107478

(remove [] from link)

This 40 message thread spans 2 pages: < < 40 ( 1 [2]
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved