Working with Googlebot - I blew it!

[peterinwa]

Much to my dismay I realized that for the past five years I have kept Googlebot from indexing all of my webpages. I'm posting my discovery here so that someone can tell me if I'm wrong.

My rank is 7 so my site gets updated frequently. My homepage has links to my major pages, so those are all indexed and show up in Google searches just fine.

The problem is with my dozens of Frequently Asked Questions and Newsletters which I post on my site.

After you go to either of these pages from the homepage, you find an introduction followed by links to all of the FAQ/newsletters. Then when you go to a FAQ/newsletter, below it you will find all of the links to the others.

I didn't want to have to update all the links with the addition of each FAQ/newsletter at the bottom of each FAQ/newsletter, so I coded the links in JavaScript. I just enter the JS code to display the links at the bottom of each page.

If you do a View/Source you don't see the links. And now I suspect that Googlebot doesn't see them either!

A number of my FAQ/newsletters are referenced by links on other, indexed pages, and these have in turn been indexed by Google. But the FAQ/newsletters that aren't linked to from another indexed page don't show up in Google searches.

Assuming the above is logical, I see two possible solutions. One would be to create a complete index in HTML so Googlebot will see it; the other is to simply enter each URL at Google myself.

Thanks for any helpful comments. I'm obviously a novice.

Peter

[tedster]

Your assumption is correct, Peter. Google will not execute javascript and so any content generated by js is not there as far as the algo is concerned.

Having the urls there in 'straight' html is the preferred solution. Even if you submit urls successfully, they will often not "stick" in the index long term, unless links are also found pointing to the pages.

[TerrCan123]

Sounds like you need a site map with all your pages too.

[peterinwa]

tedster, good point that submitting it directly to Google it might not stick. Thanks. I wouldn't have known that.

TerrCan123, my site is layed out so simply I would never have thought of the need for a site map. But that seems a good solution to the Google problem!

Hmmm. Now that I think of it, I'm endlessly bothered that people don't read my help files when they have problems. Maybe some would go to the site map (after all, why would they click on Help when they needed it!) and I could also list my Help files there. Good idea!

As an aside, one of the things I like about displaying my links via JS is that I like to make the current page link another color and in fact, not a link. Can't do this with HTML. So I'd hate to use HTML. But on a Site Map page it won't matter!

Thanks! I think this solves it!

Peter

[trinorthlighting]

Tester,

You mentioned that google does not execute java script. Now analytics uses java script and it is currently sitting in my header for one of my ecommerce sites. The pages have recently been de indexed on this site.

Could this be a cause?

[tedster]

Trinorthlighting, to be more complete I should say Google doesn't automatically execute user's javascript in order to generate page content. But no analytics js that I've seen creates part of your visible page content, and there's no reason that should cause a penalty.

If you are talking about Google's own analytics, that would be even stranger. No, I doubt that this is the problem.

[peterinwa]

I just realized I have a whole different, but related problem.

Part of my site is a food calculator for major restaurants like McDonald's. If you search Google for <a 2-word phrase> you get all my competitors, but not me! I just realized it's the same problem as discussed above -- all the restaurants are listed on the face of my calculator, and the calculator is 100% generated through JS so Googlebot doesn't see it at all!

Fixing this could make a major difference in my visitor count.

As already discussed, the answer will be too create a straight HTML list of the restaurants.

I'll be making my Site Map soon and could list them there, but wouldn't that just cause people to find the Site Map? That wouldn't help.

Another thought is that the calculator has several hidden frames. For example, the one used to generate the calculator itself. Could I list all the restaurants in a hidden frame? The code would be such that they would display if you looked at the frame, but a user never would.

I feel like this last idea is cheating Google as the list won't actually display. But I'm only trying to let Google know about my true content. Still, if I did this, would Google search results link to the calculator? Or to the hidden frame which would be pointless?

Any ideas would be greatly appreciated. I'm really excited about adding the Site Map to solve the problem of the missing links -- haha -- and now the restaurant list.

Thanks for all your help,

Peter

[edited by: tedster at 10:05 pm (utc) on May 24, 2006]
[edit reason] remove specific search terms [/edit]

[tedster]

First thing to realize is that even though a frame may display on the same "page", that is, it is visible in the same window for a user, it is really a separate document. Any content in the framed document will not be indexed as part of the parent document that frames it.

Bottom line -- if you want a given url to be located for certain search terms, then those terms must be on that url's html document. If you intentionally hide text, you nare playing with fire.

[peterinwa]

tedster, I don't understand how this works.

I have a top level URL then:

<frameset cols=160,*,0,0...

There are four frames under the top level URL.

The first is 160 wide and contains my AdSense ads. The second starts out a blank page, then is re-written as the image of the calculator when it is generated.

The first blank hidden frame holds the JS that regenerates the calculator image with each calculation; and the second holds the data for, say, McDonald's and is rewritten each time the user selects a different restaurant list.

What I'm confused about is that Google is smart enough to only show the top level frame in search results. I mean if I have the word McDonald's on a visible sub-frame, the link in Google's search results won't take you to the sub-frame which would fail to display. It always takes you to the top level frame.

I'm not making sense but do you see what I'm confused about?

I call the sub-frame the calculator is displayed in image. Image starts out blank, then the image of the calculator replaces the blank page. I'm thinking I could display my list of restaurants -- in plain HTML -- on this blank page. Then when the calculator image is generated it would step on it.

But assuming Googlebot indexes each of the sub-frames, it would get indexed.

Thanks

P.S.

I just read your post once more. It sounds like sub-frames don't get indexed at all... only the top level frames.

But when you use frames, the top level frame never displays at all, does it?

[tedster]

While any kind of frame may end up looking like "one page" when it's all assemled in a window -- in reality it is complosed of several urls. It is the url that any search engine indexes, not the composite of several urls, put together in one window. So the text and other elements on any given url only work for that url's scoring -- they don't get combined into one big page for ranking purposes.

In addition, any frame or iframe has an initial "default" configuration defined by the src= attribute. That is NOT a link, it's an attribute.

Google says that it indexs the urls found in src attributes, but I don't see them crawling such urls nearly as often as they crawl regular links. So I think it's important to always include a "straight link" to the default document in any frame. The changes in that frame are generated by a regular link, plus a target= attribute, and these urls are crawled more frequently.

I hope that helps -- at least to clarify the discussion a bit.

[le_gber]

just one idea for op, instead of using a complexe JS for your link how about using a single include file at the bottom of each of your FAQ/response. that way you only have to modify one file and it will update all your page.

[peterinwa]

le_gber, I create the list of titles and links in a .js file as a variable named "list." Then at the bottom of each of my pages:

c+=list;
document.write(c);

I don't know what "include" is but I suspect this is similar. When I edit the "list" it automatically affects all the pages.

The problem is that Googlebot doesn't see this JS coded list.

Peter

[le_gber]

peterinwa an include is a file that can be displayed on other files.

Let's say that you have an include with the following.

Copyrights c 2006 peterinwa - All rights reserved.

Now all your pages (let's say 10,000) call this include to create their footer - the text
Copyrights c 2006 peterinwa - All rights reserved. will appear as if it was part of the page and you would not be able to tell that it's an include.

in 2007 you just update the one include file and your entire website is updated once the one include file has been uploaded.

You could do the same with hard coded links instead of using a js file.

[peterinwa]

Thank you. I didn't find include in either of my old HTML or JS books, but found it on the web.

I am assuming then that Googlebot will see text that appears in an "included" file.

Peter

[pageoneresults]

There are four frames under the top level URL.

Is the site itself in frames? Or just that one area for the calculator?

[peterinwa]

The homepage #*$!.com/index.html.

The calculator is xxx.com/index_calc.html. That's the URL of the top level frame with the frameset info in it.

Thanks.