Getting Ajax content indexed

Going to start trying to get G to index my Ajax loaded content, any advise?

[Demaestro]

I have been working on a CMS that using Ajax to put content into pages.

Launched a site a while back that has only one page using the CMS content loaded via Ajax and the rest of the pages are just static HTML.

All pages rank well but for the one page the Ajax loaded content is not indexed. Instead it has some of the navigation text as the indexed content.

Since this is a playground site I am not concerned at all but I am wanting to play a little.

Any advise on getting this content indexed?

[edited by: Demaestro at 3:02 pm (utc) on Aug. 7, 2008]

[StoutFiles]

I've thought about that before, I really don't think you can. I mean, you could get it indexed as a seperate page with a sitemap but they'd be linking to a page with just that content and I'm sure that's not what you want. I specifically design pages where the ajax content is not important for indexing.

Maybe inside a <noscript>? I'd like to see the answer for this one if there is one.

[vincevincevince]

You have forgotten the Golden Rule of AJAX which is that your AJAX site must always work without Javascript. This means that it must degrade into a usable website.

A simple example is having links with both onClick going to AJAX and HREF going to http:// locations. Set up a server-side system to deliver those http pages; and add AJAX code so that if someone wonders in on one of those pages it launches the AJAX app at that point.

This is legally required in most developed countries as part of a reasonable effort towards accessibility; and has the added benefit of sorting out the SEO problems associated with AJAX.

[Demaestro]

Ya see I was thinking about the <noscript> but I would still want to load that dynamically, which would result in it not being seen.

[Demaestro]

vincevincevince

Sorry I don't quite follow all that.

I have an .html page that grabs data from a db by hitting a method that queries the database and then creates the XML on the fly.

What I am missing is, do the links I create point to the page with Ajax? or are they on the page with ajax? or Both? And what the destination of those links would be.

Is there a thread on this that you know of? I did some searching but not finding one.

[edited by: Demaestro at 5:03 pm (utc) on Aug. 7, 2008]

[vincevincevince]

Basically, the links on the AJAX page which cause you to get the new page from the database as XML should also work when you don't have Javascript by taking you to a new HTML page in which the XML content has already been converted to HTML and displayed.

In a way; you have your server act as the AJAX client, and return the 'finished' page as HTML. Each of these pages therefore has its own URL and can be indexed, bookmarked, etc.

page.html:
<a href="/nojs/get_section_343.html" onClick="return goAjax('section',343);">About Widgets</a>

You can use mod_rewrite to deal with the /nojs/get_section_343.html ... send it to a script which recognises the kind of AJAX which was intended, performs the look up serverside, and inserts it into the HTML page to return.

[Demaestro]

I understand, thanks vince

[httpwebwitch]

v-v-v hit the nail on the head there. The ideal of "Progressive Enhancement" is to make your app work perfectly well with Javascript disabled (hence no AJAX), but also to offer the enhanced, "ajaxified" version for those whose browsers do support it.

The way to do that is to always combine the "href" with an "onclick" that returns false. Once you start building with progressive enhancement, you'll be doing that so many times it'll become second nature.

However I can't endorse an architecture that requires a whole separate site structure (like, /nojs/... ) for AJAX and non-AJAX pages. You should be able to build single pages that work the same, whether AJAXified or not. This is not just for elegance in architecture - it has SEO implications as well.

For instance, whether you have AJAX enabled or not, you would request this URL:
/get_section.php

then maybe Javascript would request data from the API:
/API/section/getdetails.php?sectionid=343&format=HTML&securitychecksum=123456

But if you were JS-disabled, the page would be reloaded with the content inline, using data in the querystring:

/get_section.php?id=343

That's the content that the search engines can request and index. The non-AJAX version of a site will tend to rely on querystrings a lot.

What you end up with is a two-headed application; when adding new features, build the non-AJAX version first, then when that's finished build the AJAX-enabled functionality as a layer that obscures and replaces it. If you plan your architecture intelligently with a good API, you can leverage the same business logic to perform the functions for the AJAX and non-AJAX "versions".

Thus you use the same page/script for both versions of the functionality. It means that the same URL can be useful for those with and without JS enabled, and you don't get people bookmarking and linking to the /nojs/ site.

That is why I often recommend building content-generating modules that construct data objects in XML. When the site is working in non-AJAX mode, the output can be transformed (using XSLT) into HTML for inclusion on the rendered page, when it's something being requested via AJAX, the output can be transformed into JSON, and when it's a 3rd-party API call, you can deliver the output as raw XML. If you ever want to use the same content in a Facebook application, you can output data as FBML... ditto for CSV, PDF, RSS... and so on.

[Demaestro]

http

What you are describing makes total sense but is what I am looking to avoid.

My goal for this site is to completely separate server side functionality from HTML.

Basically I want it so every .html page has a .php or in my case a .py file that does all the thinking then passes what is required to the .html pages. Using JS my .html pages request contents and load widgets using the corresponding .py or .php file.

As in... /example.html requests XML data from /brains.py/example

/example_two.html requests XML data from /brains.py/example_two

I don't want any server side code mingled with HTML.

My problem is how to add dynamic content into an .html page without having the html file interact with my server side code.

[edited by: Demaestro at 8:22 pm (utc) on Aug. 7, 2008]

[vincevincevince]

My problem is how to add dynamic content into an .html page without having the html file interact with my server side code.

mod_rewrite works well there... although the browser request is for example.html; via mod_rewrite it can be handled by brains.py which reads in example.html (serverside), substitutes in values / text etc. then outputs it.

If you like the AJAX smooth loading and the like, that's fine; it just means that the content which loads when the page first opens will already be inserted via brains.py's pre-parsing. For a JS enabled browser, subsequent clicks on that page will be handled by AJAX; but search engines and non-JS users will pick up links to other URLs that identify the results.

If you have uniform AJAX calls, you can converge the two sets of logic directly; if you are running an AJAX-based js function "loadContent('john')" then given an alternative URL in the link of href="example_loadContent__john.html" you can then port your js into py and head it up with a bit of code which selects the right function according to the parameters mod_rewrite has parsed out (e.g. the internal rewrite could be /example_loadContent__john.html --> /brains.py?page=example&function=loadContent&datastring=john from which you know to pass 'john' as an argument to the py port of 'loadContent()' and use example.html as the base page for the insert)

[httpwebwitch]

I don't want any server side code mingled with HTML.

LOL so you're one of those people who gets upset when the meat and potatoes are touching on the plate. Amen. You'll probably want to use templates then. In PHP you'd use "Smarty" templates... I don't know what's available for Python.

Just keep in mind, you want people requesting (and bookmarking, and digging, and linking to) "example.html", not "/nojs/example.html" or "brains.py". Make sure your outward-facing URLs are uniform and consistent and your site will live happier ever after.

ciao

[Demaestro]

vince, http,

Thanks for the input you guys I really appreciate it.

My end Goal is to have a canned Ajax styled CMS that will allow thrid-party designers to alter .html files directly using FP or DW or whatever they wat without having to touch include files or anything that does thinking.

Then I will just have a set of standard widgets for designers to plug in, things like.... a Logged in Widget..... a nav widget.... and so on.

I have been finding that while many companies have no IT dept. often they have a designer on staff or on contract and I am trying to create something so that their designers can come in and make changes without my involvement. Right now with my marriage of html and python/php it isn't possible to just let them into those files with the editor of their choice and not worry about what they are doing to server side code.

Anyway thanks again. If I come up with a different solution I will post back but I have been thinking about this for a while and nothing other then similar implementations what you two are describing comes to mind.