Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

How to make Google crawl pages with a bit of AJAX content correctly

         

Conzap

8:52 am on Jan 7, 2014 (gmt 0)

10+ Year Member



Hello everyone,

I'm building a site that will utilize IP Geolocation from <external> Web Service, meaning that based on a visitors's IP(which region and country he's in) it will show different content inside a few divs around the site and a different set of inner links in the sidebar.

The thing is that I'd like to load that content after the initial page load using AJAX instead of doing it server-side since the request to <external web service> sometimes takes up to 2 seconds. I'm not sure how to make this extra content crawlable and not indexed separately as files being read by AJAX but as a part of the page they are loaded on, I hope I can get some guidance from the folk on this forums.

I've read about google's official AJAX crawling policy but only small parts of the site are loaded with Ajax and that's based on visitor's IP so I couldn't figure out how to create html snapshots for that nor that it's even worth doing it for my case. When I tested the scheme (request to <Web Service> for visitor's IP location and after page load loading content inside a div based on that) on a small discardable domain, I saw in the Webmaster Tools' "fetch as googlebot" preview that indeed the Ajax content was recognized BUT later on it was indexed as a separate link on the site.

So my question is: If I want to include ajax content in a div 1-2 seconds after page load (the time it takes to make a request to Web Service etc) how can I get google to read it as a part of the page (and not index it separately).

Maybe someone here has experience with using <Web Service> to geotarget his website and how to make google crawl it correctly and from different IPs :)

[edited by: aakk9999 at 1:41 pm (utc) on Jan 7, 2014]
[edit reason] Exemplified Web Service provider [/edit]

aakk9999

1:59 pm on Jan 8, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I don't think you can. As you said, by using #! in URL you may get that part of page to be indexed separately, but then it cannot be attributed to the page.

The only thing I could suggest is that you investigate selectively using server-side instead of Ajax if the request comes from Googlebot. But this could have other implications - such as page load time, and potentially seeing this as cloacking.

matrix_jan

3:46 pm on Jan 8, 2014 (gmt 0)

10+ Year Member



Not sure what you mean under "indexing separately". Under what url is the external content being indexed? I have a page that's in a similar situation as yours. Due to some delay of the external server I have to load some portion of the page using AJAX. I haven't had any issues with indexing.

aakk9999

7:36 pm on Jan 8, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If you load additional content on the page using AJAX after the page loads, then this additional content will not be indexed as part of the page unless you have #! in the URL. Google, upon seeing #! will perform a request for URLs with a special query string and the server should respond with a full page inclusive of AJAX content.

You need to make sure that by having #! URL, you do not have a separate (duplicate) of the URL without the #! which exists now.

There was a thread on the similar issue discussed last year:

Hashbang AJAX cloaking for non-AJAX pages, How will Google respond?
Apr 23, 2013
http://www.webmasterworld.com/google/4567288.htm [webmasterworld.com]

Google describes technical details on how to implement this here:

Making AJAX application crawable - Getting Started
https://developers.google.com/webmasters/ajax-crawling/docs/getting-started [developers.google.com]