why has google indexed specific pages of my site and not others?

[BigMrC]

I submitted a website to google that contains frames. When you search for this site under its name 2 pages are found neither of which is my main page. When you then choose "search within ommited results" it brings up all the rest of my pages but there's no cache info for them. Why did it choose the 2 pages it did to display? It didn't even cache my main index page

[troels nybo nielsen]

Welcome to WebmasterWorld, BigMrC

I uploaded an article that stretches over three pages. Right now page 3 is #9 on a specific keyword, while pages 1 and 2 are non-existant in Google. Problem is they have been spidered by Googles freshbot (read about in site search) and thus live an unsecure life. When they get deepcrawled they in due time will have a more stable position in Google's search results.

[BigMrC]

ah, so are you saying leave everything as is and at some point all pages will be cached properly?
if so does this also mean that when people are searching for my site the actual index.htm will come up first rather than buried away amongst all the other frame pages? or is there something specific I can do to make sure the main index.htm gets priority over all the other pages in search results?

thanks for your help.

[troels nybo nielsen]

I must admit knowing very little about frames. I don't like them for a number of reasons, one of them being that search engines used to have problems indexing them properly. I believe that it is still very important to use frames correctly if you want all your pages indexed.

I think we will have to wait for an answer from someone knowing a lot about frames. But one thing I can say: It pays to be patient when you're dealing with search engines!

PS. It might help to know how old your website is and when you submitted it to Google.

[Marcia]

With a site in frames, make sure to have a <noframes> section that includes links to the other site pages. Frames are generally problematic where search engines are concerned, and not just Google.

[jimbeetle]

BigMrC,

Frames present special problems for spiders -- and you working with them. You can direct spiders to individual pages two ways: Put links to the pages in the <noframes> section of the frames page; or, better, build a site map that is directly accessible to the spiders.

After you get all the individual pages listed in the SEs you then have the problem of your pages coming up "naked" without the supporting frameset. You've got to include a bit of javascript that says something like "If this page loads and its parent is not on of my frames then put it in this frame." You can find a few versions on any of the javascript library sites.

You might also consider making your index page one that does not use frames. This makes it easily accessible to spiders, you can link from it to a site map and you can directly link from it to your most important pages.

We used frames for more than four years on one of our bulkier sites and built it to be completely spiderable and bulletproof -- stand alone index page, site maps, dynamic framesets, javascript frameset calls -- but found that it just wasn't worth the hassle. We've just spent almost a year changing everything to stand alone pages.

Jim

[BigMrC]

thanks for that, jimbeetle in particular. I just read on another forum that google (and yahoo?) use freshbots as well as deep searching bots to index your site. As I only submitted the site 2 days ago when it went live I wonder if its only been visited by the initial bots? I've no idea how this works but do you think it might be worth holding off changing anything for a week or two to see if the main index.htm gets indexed properly?

[jimbeetle]

Google is deep crawling now and will update its index sometime at the end of the month. It *usually* takes 2 full cycles for sites to be included in the index so don't get your hopes up for this time around (but you never know).

In the meantime make sure that there are spiderable links to all of your pages and that you build some incoming links from other sites. Those are your two best bets for getting the site and all of your pages indexed.

Don't worry about Freshbot at all right now. You might happen to pop into the index for a day or two but as its name suggests it wants pages that are fresh, so you'd more than likely drop out again.

Jim