homepage Welcome to WebmasterWorld Guest from 54.211.230.186
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
New Domain Not Being Indexed
Uber_SEO

5+ Year Member



 
Msg#: 4138339 posted 12:52 pm on May 25, 2010 (gmt 0)

About 2 months ago we launched a couple of new sites on subdomains. We have a very established main domain, and we linked to both subdomains from the main navigation on the main domain. One of these subdomains was a blog - we did some PR for it, and a campaign on Twitter, and the blog ended up being indexed very quickly.

However, 8 weeks on, and the other subdomain still hasn't been indexed. I've audited it for technical issues - it has no robots.txt, no robots meta tags, and always returns correct HTTP codes. It has links from our own group of sites, but very few from elsewhere. I've created a GWT account, which suggests there is very little wrong with it. I've also submitted for Google for reconsideration, who have given me the usual spiel about checking their guidelines etc...

So in summary, I've 99% sure there is nothing technically wrong with this website. Anyone else seeing any similar behaviour from Google at the moment?

 

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4138339 posted 4:24 pm on May 25, 2010 (gmt 0)

I've created a GWT account, which suggests there is very little wrong with it.

That seems to indicate that at least some URLs are being indexed - right? Webmaster Tools only has data for indexed URLs, so you might mean there's a low level of indexing, or that the site: operator number is bad, or...

Uber_SEO

5+ Year Member



 
Msg#: 4138339 posted 6:15 pm on May 25, 2010 (gmt 0)

Sorry Tedster - should have been more specific. The site definitely isn't indexed - you won't find it in Google by doing a cache: search, a site search, a search for the domain name, a Toolbar cache, or if you search for a snippet of homepage text. It's definitely not there.

Weird thing is, server logs are showing Googlebot accessing the website and seeing 200 status codes. Even crawl stats in GWT is showing some (albeit small) level of Googlebot activity on the site. If I fetch as Googlebot in GWT then everything works fine. Other than that, there's no data available within GWT.

internetheaven

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4138339 posted 6:40 pm on May 25, 2010 (gmt 0)

I've had 2 sites not get indexed for over a month now. However, 11 other sites launched at the same time have all been indexed fine and are even ranking for various long tails.

I can't tell the difference between the 2 that are not being indexed and the 11 that have, other than the 2 that haven't been indexed have many more backlinks ... which would suggest faster indexing wouldn't it? It doesn't make any sense.

All 13 sites had noindex tags for about 2 months whilst being developed. The noindex tags where all removed at the same time.

Uber_SEO

5+ Year Member



 
Msg#: 4138339 posted 7:50 pm on May 25, 2010 (gmt 0)

@internetheaven - thanks for the reply. the subdomain that i'm having problems with initially had a noindex tag, and all links to it were nofollows. We dropped these though when wanted to get the sites indexed. I'm wondering whether there's a bug with Google whereby if a link / page was nofollowed / noindexed, and that gets changed, then perhaps Google keeps the old nofollow / noindex status assigned to it

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4138339 posted 8:35 pm on May 25, 2010 (gmt 0)

To clarify - in addition to a robots.txt disallow directive that you mentioned above, did you also use either a robots meta-tag nofollow or a nofollow attribute at the individual link level?

nickyo

10+ Year Member



 
Msg#: 4138339 posted 8:53 pm on May 25, 2010 (gmt 0)

@internetheaven - I've had the same thing with a site not being indexed. Launched 2 sites about 2months ago, indexed fine, then another about 4wks ago, still not showing up. No reason why it shouldn't be, content is good, links in moderation etc. Should be fine, but still not appearing.

seoN00B



 
Msg#: 4138339 posted 6:10 am on May 26, 2010 (gmt 0)

id experience this one before. When i created forum (simple machines).
It takes 4 weeks to get indexed... it was kinda weird.

Uber_SEO

5+ Year Member



 
Msg#: 4138339 posted 2:18 pm on May 26, 2010 (gmt 0)

@tedster - it was a page level noindex, follow meta tag, and external links from our main domain were nofollowed. Both have now been removed

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4138339 posted 5:54 pm on May 26, 2010 (gmt 0)

That means a lot of pages need to be re-crawled - and possibly more than once, to confirm - before they are included in the index. Four weeks does seem just a bit slow, but it's not outside other's experience.

Uber_SEO

5+ Year Member



 
Msg#: 4138339 posted 10:59 am on Jul 1, 2010 (gmt 0)

Since I last posted here, the domain has still not been indexed, despite being crawled by Google. Any other ideas?

centime

5+ Year Member



 
Msg#: 4138339 posted 12:42 pm on Jul 1, 2010 (gmt 0)

@tedster

Does that mean that google crawls a complete site before the pages start to showe up in the index ?

Uber_SEO

5+ Year Member



 
Msg#: 4138339 posted 2:35 pm on Jul 1, 2010 (gmt 0)

@centime - I don't think that should be the case. Google should index the pages as it finds them - no need to crawl the whole site before it indexes it. There's definitely something really strange happening with the domain

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4138339 posted 5:43 pm on Jul 1, 2010 (gmt 0)

@tedster Does that mean that google crawls a complete site before the pages start to showe up in the index ?

No that has never been the case.

There's definitely something really strange happening with the domain

Sure sounds like it. This is a subdomain, correct? Is it on a dedicated IP?

I've never personally taken an indexing problem to Google's webmaster support forum. But in this kind of situation, I would at least consider it.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved