New Domain Not Being Indexed

[Uber_SEO]

About 2 months ago we launched a couple of new sites on subdomains. We have a very established main domain, and we linked to both subdomains from the main navigation on the main domain. One of these subdomains was a blog - we did some PR for it, and a campaign on Twitter, and the blog ended up being indexed very quickly.

However, 8 weeks on, and the other subdomain still hasn't been indexed. I've audited it for technical issues - it has no robots.txt, no robots meta tags, and always returns correct HTTP codes. It has links from our own group of sites, but very few from elsewhere. I've created a GWT account, which suggests there is very little wrong with it. I've also submitted for Google for reconsideration, who have given me the usual spiel about checking their guidelines etc...

So in summary, I've 99% sure there is nothing technically wrong with this website. Anyone else seeing any similar behaviour from Google at the moment?

[tedster]

I've created a GWT account, which suggests there is very little wrong with it.

That seems to indicate that at least some URLs are being indexed - right? Webmaster Tools only has data for indexed URLs, so you might mean there's a low level of indexing, or that the site: operator number is bad, or...

[Uber_SEO]

Sorry Tedster - should have been more specific. The site definitely isn't indexed - you won't find it in Google by doing a cache: search, a site search, a search for the domain name, a Toolbar cache, or if you search for a snippet of homepage text. It's definitely not there.

Weird thing is, server logs are showing Googlebot accessing the website and seeing 200 status codes. Even crawl stats in GWT is showing some (albeit small) level of Googlebot activity on the site. If I fetch as Googlebot in GWT then everything works fine. Other than that, there's no data available within GWT.

[internetheaven]

I've had 2 sites not get indexed for over a month now. However, 11 other sites launched at the same time have all been indexed fine and are even ranking for various long tails.

I can't tell the difference between the 2 that are not being indexed and the 11 that have, other than the 2 that haven't been indexed have many more backlinks ... which would suggest faster indexing wouldn't it? It doesn't make any sense.

All 13 sites had noindex tags for about 2 months whilst being developed. The noindex tags where all removed at the same time.

[Uber_SEO]

@internetheaven - thanks for the reply. the subdomain that i'm having problems with initially had a noindex tag, and all links to it were nofollows. We dropped these though when wanted to get the sites indexed. I'm wondering whether there's a bug with Google whereby if a link / page was nofollowed / noindexed, and that gets changed, then perhaps Google keeps the old nofollow / noindex status assigned to it

[tedster]

To clarify - in addition to a robots.txt disallow directive that you mentioned above, did you also use either a robots meta-tag nofollow or a nofollow attribute at the individual link level?

[nickyo]

@internetheaven - I've had the same thing with a site not being indexed. Launched 2 sites about 2months ago, indexed fine, then another about 4wks ago, still not showing up. No reason why it shouldn't be, content is good, links in moderation etc. Should be fine, but still not appearing.

[seoN00B]

id experience this one before. When i created forum (simple machines).
It takes 4 weeks to get indexed... it was kinda weird.

[Uber_SEO]

@tedster - it was a page level noindex, follow meta tag, and external links from our main domain were nofollowed. Both have now been removed

[tedster]

That means a lot of pages need to be re-crawled - and possibly more than once, to confirm - before they are included in the index. Four weeks does seem just a bit slow, but it's not outside other's experience.

[Uber_SEO]

Since I last posted here, the domain has still not been indexed, despite being crawled by Google. Any other ideas?

[centime]

@tedster

Does that mean that google crawls a complete site before the pages start to showe up in the index ?

[Uber_SEO]

@centime - I don't think that should be the case. Google should index the pages as it finds them - no need to crawl the whole site before it indexes it. There's definitely something really strange happening with the domain

[tedster]

@tedster Does that mean that google crawls a complete site before the pages start to showe up in the index ?

No that has never been the case.

There's definitely something really strange happening with the domain

Sure sounds like it. This is a subdomain, correct? Is it on a dedicated IP?

I've never personally taken an indexing problem to Google's webmaster support forum. But in this kind of situation, I would at least consider it.