Google won't index pages

Only some pages indexed

[gefilte]

Sorry for the newbie question, couldn't get a good answer from FAQ:
My site has been on Google for 8 months. The top page is included but only a few of the others. The others that are indexed are linked from other sites.
My structure is three levels: top page, index page, content pages. All pages have backlinks. The index page is also linked from another site, but Google refuses to index it. Is this it? Will Google ever index it or the other content pages?

[OZZY2662]

Hi gefilte,

Feed the spider. Add links from the main page to below pages.

[jamesyap]

Build a site map page too that link to every pages in your site.

[gefilte]

Thanks for the advice, still a little unsure.
The top page links to the index page, yet Google has never added the index page, how will adding more links on the top page help? Won't they just be ignored too?
Essentially my index page is a site map with links to every other page, unless there is something I don't get about a site map.

[jamesyap]

Do you do anything tricky (spamming?) that causes your domain to be banned? Do you link to a link farm/banned sites? You may search on the forum on 'penalty'.

[gefilte]

No, and my site is listed and gets freshbot every day. Five of the content pages are listed too. The index page and 60 some content pages are not. This seems strange, shouldn't the whole thing get deepcrawled at some point?

[percentages]

gefilte, this makes no sense. If your pages are of the type Googlebot can understand then they should be indexed.

How are you determining they are not indexed? Does a search for a specific page show no results? Are you using the Google Toolbar to determine if they are indexed (it is largey a guess and pretty worthless IMHO).

My suggestion is you add a single word to each page that you think is not indexed. Make the word totally unique to your site, something really obscure. Then wait for a couple of Google updates and see if it found the pages on the obscure word ;)

[gefilte]

Yes, I know it makes no sense, that's why I'm asking. I know they're not indexed because I search for each specific page. I also have a unique word in all my pages that I can search for, I check every page that comes up, only sites that link to me and the few pages indexed. I appreciate you all trying to help me figure this out very much.

[gefilte]

Here it is again, another update and still only 6 of my pages are in the google index. What else can I do to get the rest of the pages spidered?

[daamsie]

Hi gefilte,

Your site seems to not be breaking any rules as far as I can tell (if it has anything to do with parodies, that is...).

I couldn't check the PR of your pages directly, because I'm on a mac and no chance of a toolbar, but you have a 5 judging by the directory, so I imagine the 'index' page is a 4..

One query I have is this: does any conflict occur with the googlebot when it finds an 'index.html' page that is not the default page of the website.. perhaps the bot gets confused or sees this as a possible spam issue. It may be worth renaming that page to something a little less common, so as to eliminate that possibility. Just guessing, because like you, I don't really understand the problem.. Maybe a more experienced SEO could enlighten us on that one :)

I would suggest including meta tags and spelling it out for the robots a little more though.. your index page doesn't have any metas at all, which doesn't help your cause..(particularly in other SEs) Of course, that shouldn't stop Google from indexing them either..

[gefilte]

Thanks daamsie,
I was under the impression however, that Google ignored Meta tags. Would putting them on the index page make any different? I still don't see why google hasn't included my index page, much less the rest of the site.

[OZZY2662]

I had a similar problem with not crawling beyond the home page. I renamed the index.html page and inserted links on the home page to the site map page and other pages. On the next month Deep Crawl the spider crawled through the hole site.

[daamsie]

Hi Gefilte,

Although Google ignores meta tags in its results (ie, it doesn't factor them into its algorithm), it does pay attention if you tell it not to index your page or not follow links etc.. it does pay attention to the meta robot tag..

This doesn't explain your problem though, because the default meta robot setting is index and follow.. I just thought it wouldn't hurt to add it and the keywords and description tags as well.. might help you get listed in other SEs out there.

Seems to me though the thing that is most likely causing problems is the naming of your files.. index.html is usually the default page of a website, so it wouldn't surprise me if using that page as a non-default page creates a problem in Google's eyes. I don't know, but seeing as Ozzy had a similar problem, I think it would be worth trying.