Welcome to WebmasterWorld Guest from 54.162.44.105

Forum Moderators: Robert Charlton & andy langton & goodroi

Message Too Old, No Replies

Matt Cutts: Googlebot Can Now Execute AJAX/JS

     
7:23 pm on Nov 1, 2011 (gmt 0)

Administrator from GB 

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month

joined:May 9, 2000
posts:24136
votes: 518


According to Matt Cutts,Googlebot can now execute AJAX/JS to index some dynamic content.

[twitter.com...]

This is big stuff.
7:41 pm on Nov 1, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


In particular, Google can now index Facebook comments as well as those on some other social sites such as Disqus or Intense Debate. See http://www.techmeme.com/111031/p54#a111031p54 for more detail.

(Note: the fragment identifier gets broken if we post a live link - so please copy/paste.)
8:09 pm on Nov 1, 2011 (gmt 0)

Junior Member

5+ Year Member

joined:Nov 25, 2009
posts: 42
votes: 0


This is really interesting.

Are they going to base serps on what is trending in comments and of course all of the other variables (location, etc)?
8:15 pm on Nov 1, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member wheel is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Feb 11, 2003
posts:5072
votes: 12



In particular, Google can now index Facebook comments

OK, so you're going to talk about this next week at pubcon then? Or someone's got to!
8:42 pm on Nov 1, 2011 (gmt 0)

Preferred Member

5+ Year Member

joined:Dec 19, 2007
posts:404
votes: 0


wouldn't they need to be a friend to read your fb comments? :)

[edited by: santapaws at 9:06 pm (utc) on Nov 1, 2011]

8:48 pm on Nov 1, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


As I understand it, this is about the Facebook Comments Plug-in [news.softpedia.com] that's in use on many websites.
3:04 am on Nov 2, 2011 (gmt 0)

New User

5+ Year Member

joined:Nov 2, 2011
posts: 8
votes: 0


The ex-Googler @pedrodias tweeted about it a couple days ago.
[twitter.com ]

It's realy a good news.
3:31 am on Nov 2, 2011 (gmt 0)

Preferred Member from TH 

10+ Year Member

joined:Mar 4, 2003
posts:422
votes: 0


Big move - Google will crawl more contents and links. But we need to keep moderate the links on comment (this will create spam opportunity to spammers). Can we set up no-follow links on Facebook comment box?
10:32 am on Nov 2, 2011 (gmt 0)

Junior Member

5+ Year Member

joined:June 7, 2011
posts: 97
votes: 0


I think this is really interesting too. I wonder if Facebook will put a stop to it asap. Loophole or intentional? Facebook does not want Google on their side at all, just look at their robots.txt
12:55 pm on Nov 2, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 16, 2003
posts:992
votes: 0


Facebook does not want Google on their side at all, just look at their robots.txt

Those are pretty sensible exclusions, preventing the indexing of search pages, duplicate content, and so on. Google reports over 5.6 billion results from FB in its index.

This is good news for FB if if encourages wider adoption of its comment system.
1:55 pm on Nov 2, 2011 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3236
votes: 195


Be careful to keep this in perspective.

Matt said Google can
"...index some dynamic comments..."


Matt did not say Google can index ALL dynamic comments. Webmasters should still be careful when adding AJAX to their site.
1:56 pm on Nov 2, 2011 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 30, 2002
posts:4861
votes: 11


Likely a case of 'it can parse well known frameworks' to varying extent rather than "ajax/js".
2:03 pm on Nov 2, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:12929
votes: 200


I'm glad to see it. Fits in quite well with my evil empire.
2:10 pm on Nov 2, 2011 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14663
votes: 99


Skynet could also execute AJAX, hmmmm....

OMG! It's indexing Facebook to find Sarah Connor!

[edited by: incrediBILL at 2:23 pm (utc) on Nov 2, 2011]

2:18 pm on Nov 2, 2011 (gmt 0)

New User

5+ Year Member

joined:Nov 2, 2011
posts: 8
votes: 0


Big move - Google will crawl more contents and links. But we need to keep moderate the links on comment (this will create spam opportunity to spammers). Can we set up no-follow links on Facebook comment box?


I think that all links on facebook comments are no-followed and I think you can't change this!
2:33 pm on Nov 2, 2011 (gmt 0)

Junior Member

5+ Year Member

joined:Jan 22, 2011
posts:96
votes: 0


In particular, Google can now index Facebook comments as well as those on some other social sites such as Disqus or Intense Debate. See [techmeme.com...] for more detail.


I use Wikipedia text in some of my image heavy pages, but I load the text through AJAX so bots can't see it. Could this now possibly affect my rankings, if G is able to load that text?.
3:00 pm on Nov 2, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2005
posts:2259
votes: 0


The big danger here is that Google is now going to index some of the data we thought we had blocked it from crawling.

From the GorgBlog [googlewebmastercentral.blogspot.com]

Google crawls the URL, yummy-sundae.html.
Google begins indexing yummy-sundae.html and...decides to attempt to render the page.
During the render, yummy-sundae.html automatically sends an XMLHttpRequest for a resource, hot-fudge-info.html, using the POST method.
6:53 pm on Nov 2, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:12929
votes: 200


I use Wikipedia text in some of my image heavy pages, but I load the text through AJAX so bots can't see it. Could this now possibly affect my rankings, if G is able to load that text?.


If I understood this video from Matt Cutts, something like that would probably be considered cloaking, which could possibly get you into trouble.

[youtube.com...]
8:13 pm on Nov 2, 2011 (gmt 0)

Junior Member

5+ Year Member

joined:Jan 22, 2011
posts:96
votes: 0


If I understood this video from Matt Cutts, something like that would probably be considered cloaking, which could possibly get you into trouble.


Well that sounds scary. The way I understood the video is that if you target Googlebot and present it with different data, then that would be considered cloaking. I'm basically showing my pages this way to all users with javascript enabled.

Edit
Just saw the last part of the video. Guess I'll be changing my pages.
8:53 pm on Nov 2, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:12929
votes: 200


Probably the safest way to go.
12:02 am on Nov 3, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member sgt_kickaxe is a WebmasterWorld Top Contributor of All Time 5+ Year Member

joined:Apr 14, 2010
posts:3169
votes: 0


This is old news, Google's been indexing words that do not appear on page if you view source for over a year now. I suppose they had to make it official though.
12:35 am on Nov 3, 2011 (gmt 0)

Preferred Member

10+ Year Member

joined:July 12, 2004
posts:384
votes: 0


I noticed Google indexing my Ajax content more than a year ago.

So, if anyone was thinking ajax was a way of hiding content from google then it has been a no-go for ages.

I totally hide anything I don't want indexed by Google - that is stuff like ads, dupe content etc. - and yes it is cloaking, but I simply don't care if there is an issue - I need to pay the bills for my family.
12:39 am on Nov 3, 2011 (gmt 0)

Preferred Member

10+ Year Member

joined:July 12, 2004
posts:384
votes: 0


In fact, I have started doing anything that works - forget Google guidelines etc. or if it is "ethical" and you know what, it pays better than a strategy that looks at ranking well in Google.

Over the past year I have shifted to simply thinking about users and then manipulating Google as I can to get traffic to my content.
12:53 am on Nov 3, 2011 (gmt 0)

Junior Member

5+ Year Member

joined:Jan 22, 2011
posts:96
votes: 0


I only added the ajax stuff yesterday, that's why it's easy for me to undo. But, yeah one of my competitors has been using that since ever and is ranking well.
4:18 pm on Nov 3, 2011 (gmt 0)

Junior Member

5+ Year Member

joined:Jan 22, 2011
posts:96
votes: 0


Well, I've decided to put the Wikipedia snipped back. I'm not loading the text through JavaScript this time.
4:36 pm on Nov 3, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 17, 2006
posts: 838
votes: 0


They've been executing JS for quite some time now (3-4 years I think) to fish for hidden links outside HTML code. I understand that could be done during indexing, i.e. not in real time.

But are they now executing it during crawl or still during indexing/processing stage? I mean, those Facebook comments and the like are all time-sensitive info. It would make no sense to execute THAT JS later - so are they going to run all JS on all pages they download right that instant?

I did not think there's enough computing power in the world that would let anyone do that, not at their scale anyhow. How about JS with errors? Endless loops and otherwise bad code? Wow, this looks like a can of worm I would prefer to keep sealed ...
12:41 pm on Nov 4, 2011 (gmt 0)

Junior Member

5+ Year Member

joined:Nov 26, 2009
posts:90
votes: 0


This is, as others have said, hardly new news.

So long as the link contains hard URL data, Google will be able to find it, and has been for some time. I've seen sites, for example, who's JS links contain just a string to add to the end of the URL, for example "parameter=filter1", and Google has tried sticking it on the end of that pages URL and spidering the result.

Rather handily, when its a navigation element, Googlebot seemed to get stuck in a loop for a bit, resulting in it spidering pages that had the query "Parameter1 + Paremeter1 + Parameter1 + Parameter1".
1:22 pm on Nov 4, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 31, 2001
posts: 1357
votes: 0


@dududoseo
I think that all links on facebook comments are no-followed


So you think that Googlebot doesn't follow no-follow links ;)
9:21 pm on Nov 4, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13674
votes: 439


In practical terms... I noticed that the major search engines have started picking up javascript pages. Fine, if it's intrinsic to the page content. But how the ### do you keep them from picking up things like piwik code that they have no business snooping around in?

:: not happy about the prospect of manually changing a bunch of links to go along with relocating code snippet to roboted-out directory where it doesn't really belong ::
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members