Include files, spiders, and robots.txt

[rottsnroses]

Hi ya'll. I have a question on using include files, search engines, and robots.txt.

I know that the closer keywords are to the top of the page, the better it is for search engine rankings. I have graphics and javascipt navigation at the top of my site in an include file, before the page content, since it is used on every page. What is in the include file really has no value to the spiders, and only enhances the appearance of the site.

My question is if I disallow the directory where my include files live to the robots.txt file, will that result in the spiders skipping that part of the page, and going directly to the content, resulting in my keywords being at the top of the page, rather than the graphics?

[Dijkgraaf]

It depends on how you are doing the include.
If the include is server side, then the bot won't even know the source directory of the include file.

If you are including external JavaScript, most bots won't even request that, but it wouldn't hurt to exclude them from that folder if it doesn't containt anything usefull for bots to look at.

Most bots ignore graphics anyway, so I don't think telling bots not to request things from the graphics folder would improve your ranking. If you don't want certain bots that catalog images to get hold of them, or you are worried about bandwith getting used up, then yes, dissalow the graphics folder.

[rottsnroses]

Perfect! That is just the answer I needed to know.

[Dijkgraaf]

One thing about externally included JavaScript.
If you look at a cached version of a page, Google sets the base URL of the page to your web site, so that externally loaded JavaScript will still actually load and run in the cached page.
If you don't want that you would have to a routine in the JavaScript that would check to see document.location matched your domain.