Why has Google indexed one of my new sites but not another?

Problem using a domain name used on other sites?

[HarryM]

A few weeks ago I split off certain pages from an exisiting site and created two new sites. The three sites are on the same server (in the UK) and the old pages have been deleted. Both new sites have an incoming link from the index page of my old site and there are no 301 redirects.

One of the new sites (the one I am least bothered about, natch!) had all its pages indexed by Google within a few days. But with the other site, although Google visits at least once per day, it only takes robots.txt and the index page. It re-caches the index page daily but never goes deeper.

This is the opposite of what I would expect because the site that is crawled is just parked with never any changes, whereas the site that is not crawled has frequent updates to its index page, new pages added daily, and at least one incoming link from another site that Google knows about because the link shows in Google's cache of that page.

The only difference that I can see is that the crawled site has a unique domain name, whereas the uncrawled site has a domain name that is used in other sites. Mine is www.placename.com, but there is also www.placename.co.uk, www.placename.org.uk, and www.placename.countyname.co.uk.

My index page is fully indexed, but the other sites just have urls listed. Which is not surprising because www.placename.countyname.co.uk. seems to be an unused subdomain, www.placename.co.uk displays only an error message, and www.placename.org.uk is under construction and uses frames.

Could the domain name be the cause of the problem? Or any other suggestions?

[HarryM]

Another couple of days have gone by with no change. Google still indexes and re-caches the index page every day but goes no deeper.

I've got link requests and a DMOZ entry in process so I don't want to buy another domain name unless absolutely necessary.

I'm baffled - does anyone have any suggestions?

[HarryM]

Thought I'd update this just to see if I can get a reponse.

Last night Google again indexed and re-cached the index page but has gone no deeper.

There can't be anything wrong with the site becase MSN (preview) has indexed most of the pages.

Does Google have problems with sites with same domain names but a different TLD?

[DerekH]

The stock reply, and one that it's definitely worth taking on board is to feed the errant page through the w3c validator and check that there's not a syntactical error in it - syntax errors are often invisible in a browser, which can take time patching up the syntax because the display time is so much longer than the fetch time. Robots are in a hurry, and if they get confused, they'll drop the page and move on.

And there are robots.txt checkers too.

Be sure you've check the pages thoroughly.
It's certainly the case that Google seems to gradually drill down into a site - I frequently find that new sites appear in stages - the index page and perhaps one or two more first... Then quite a sizeable delay before the rest appear.
DerekH

[HarryM]

Thanks DerekH, but the first thing I did was validate the pages and check robots.txt. I also ran the index page and a few others through simspider. Also the MSN bot has no problems indexing the site.

The site went up on 24 November, and a second site on 26 November. The second site was fully indexed within a few days, but only the index page of the first site is indexed. Google re-caches this index page daily but never goes deeper.

It may be just a fluke, but then again it may not, so how long do I wait? The only difference I can see between the two sites is that the unindexed one uses a domain name that is in use with other TLDs. I would like to know if anyone has found this to be a problem.

[DerekH]

Sorry Harry if I stated the obvious, but unlike you, many don't bother to check what they've done <grin>

I run one site for a local charity, comprising a domain name with URL forwarding, and the site sitting on a free ISP.
For reasons irrelevant to this discussion, I moved the entire site and re-pointed the domain name to the new site on 5th December.
The domain name seems to get visited at 4 in the morning every single day, but it took 3 days to get the first page of the new site into the index, and that's all that's happened.

I'm not at all worried though - it's only really high ranking pages that get high ranking service - I'll not be expecting other pages to make the index for about 2 weeks now - the further into the site the lower the PR, and the more infrequent the visits. Sometimes the whole site might be fortunate enough to burst into life in the index, but I put my success rate at about 50% for that - the other 50% just slide in, bit by bit. And though I can find no reason why this behaviour might be so polarised, nor any factor to influence, I'm old enough and ugly enough to know that the best I can do is get on and write content instead!
DerekH
A watched pot never boils, and a watched site seldom bubbles up...

[irishaff]

Hi Derek,

My recent observations are that higher PR pages are getting visited less, or certainly cached less than my lower pr ones. Doesnt make sense to me , but there ya go..

[cabbie]

I would try to verify if the name has been used before.If the domain had expired within the last 12 months then it maybe under a expired domain penalty.
The other thing perhaps is that google feels no need to go deeper on your site as it attributes your site as being the equivalent of ones that you already have.It does this by the common pattern of links and having the same host.Just a thought.

[HarryM]

Cabbie, I had a look at an archive site and could find no entries, nor any old entries in the search engines. The linking structure and almost all the directory names are different to my other sites.

Anyway I get my logs weekly on Sundays so I'll be up in the small hours to see if there has been any Google activity. Keeping my fingers crossed. :)

[MHes]

Have you tried linking to a deeper page from another site? It may kick start things.

[cabbie]

>>>The linking structure and almost all the directory names are different to my other sites.<<<

Didn't you say "Both new sites have an incoming link from the index page of my old site"?
This could be a flag.Both sites are about the same topic with the same inbound link to it and on the same server.

[HarryM]

Cabbie, thanks for the suggestion, but the sites are very different. As to being on the same server there must be countless sites like mine that run on 'multiple site' hosting packages so I wouldn't envisage that to be a problem. If you want to take a look the main site is in my profile with two image links at the bottom of the page.

Just finished downloading my log files. The usual daily visit to the index page, but nothing else. Bummer...

[added later] Put Adsense on all pages. Well, it can't hurt... [/added]