Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

New site, slow to get indexed



2:56 pm on Sep 12, 2007 (gmt 0)

5+ Year Member

Hi all, long time lurker, first time poster. Sorry for the length, I'm just trying to include all information that may be relevant up front.

I recently launched my first site. After watching over this and other forums as well as 'optimization' tips, I believe I have everything laid out properly - keywords in page title and H1/H2 tags, meta descriptions, sitemap page, you name it. I would not consider my pages to be 'over-optimized' either - I'm not keyword stuffing, just following the best practices that are commonly suggested.

The only thing out of the ordinary is that that I am building my site with regional subdomains which contain the bulk of my content, such as "northeast.widgets.com". The main page (www.widgets.com) is simply one page with intro text and links to the subdomains available.

About three weeks ago I added my subdomain site to Google. I currently have about 65 pages on this subdomain. Within a week Google had picked up a handful of pages (4 or 5). At the suggestion of many on this board, I created a sitemap.xml.gz file for my site and added it to Google's webmaster tools. Late last week Google had indexed only 15 of 65 pages, despite them having the sitemap file (which Google had parsed and found no errors). This hadn't changed up through yesterday - I still had only 15 of 65 pages indexed.

This morning I checked again and it seems that I am now down to only 9 indexed pages! My site hasn't ever been fully indexed yet Google is dropping pages on me. The ONLY thing I've done in the past week which I can think may have impacted anything is that a few days ago I deleted the unique google html file that webmaster tools asks you to add for authentication purposes. I thought it was a one-time verification thing and didn't realize it had to stay there permanently. Yesterday I noticed my site in webmaster tools was back at 'unverified' so I put the file back. I don't see how that would trigger a removal of links from the search engine. One other thing I did (which I thought may have been what was causing Google to only index 15 pages) was that webmaster tools found 4 missing/404 pages - I fixed those two days ago, but it doesn't seem Google has attempted to rescan those links yet.

For what it's worth, the only 'crawl' date I can find is for those 404's which indicate August 31. So as best I can tell, my site hasn't been indexed since the 31st yet Google decided to drop some pages since the last crawl.

So, my question for the hive mind is:

- Why would Google drop pages from its engine, even when not all links had been indexed?
- Does anything in my website structure, or the changes I mention being made, have any bearing on what has happened?
- Is there any kind of fatal flaw in my setup? (subdomains containing majority of the content, etc)
- Any additional suggestions on getting all pages into the Google database?

Thanks in advance for your help - this board is a great resource for us 'unseasoned' folk! :-)


6:12 pm on Sep 12, 2007 (gmt 0)

5+ Year Member

I just made a similar comment elsewhere, but in my view it is the lack of incoming links to your new subdomain. So, I am repeating once more :)

Get an average (PR5+) or strong back-links (PR7+) from older sites linking to your subdomain and you will be get fully indexed within couple of days (depending on the competitiveness of XYZ pages).

My observations are that for non-competitive, non-commercial phrases (lets say around 1,5 mio - 2 mio of total results) couple of back-links from PR6+ long time ago established websites will make my targeted pages into top 5 SERPs usually within 3 days. PR8+ back-link made it within 10 hours...

From a technical point of view all your other steps taken seems to be OK. Due to the lack of backlinks, your subdomain seems to be unimportant to spiders and there they do not bother to index it.


6:32 pm on Sep 12, 2007 (gmt 0)

5+ Year Member

in my view it is the lack of incoming links to your new subdomain

Highly likely that this is the cause. Sites with lots of links, especially those of high value, get attention and are indexed quickly.

Unfortunately, quality backlinks just donít grow on trees.


6:37 pm on Sep 12, 2007 (gmt 0)

5+ Year Member

Thanks for your feedback. Seems kind of like a chicken-or-egg scenario, though. Certainly no one would find my site and link naturally, and it seems it would be rather difficult to get a well-established site in my niche to link to me as I'm a brand new site with hardly any links showing in the search engine.

I'll definitely look into this though.. I was just hoping my site could (at the very least) get its foot 'in the door' on its own merits rather than having to beg one of the 'big guys' to link to me just so Google will even bother to look at me.


7:27 pm on Sep 12, 2007 (gmt 0)

10+ Year Member

<<Certainly no one would find my site and link naturally>>
...so... buy some good link


8:18 pm on Sep 12, 2007 (gmt 0)

5+ Year Member

Yea it probably is worth buying them but I would n't advise this as a way forward for too long...


10:36 pm on Sep 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Be patient. A new site like yours could easily take 6 weeks to get indexed.

As for 'dropping' pages, that's just flux. They'll be back in a few days.

Try to get a few links, but I would strongly advise you stay away from the paid route, or any spammy directories. That will likely just open worse problems.

Go do something else, and let it cook. Put a little time into getting links, but don't sweat it.


10:45 pm on Sep 12, 2007 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

As with many businesses, start-up for a website can involve marketing of some kind. Legitimate directory listings, social media - really finding any way you can to let interested parties know that there's a new kid in town.

You're right, just sitting there and getting a few backlinks from scraper sites can be tough start-up. However, at some point early on, Google often gives new sites a boost in the SERPs for a day or two - I've called it the honeymoon in the past. That might help spread the word, too.


3:11 am on Sep 13, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I went public with my new site on May 25th. It took several weeks before Google had indexed 75% of the pages on the site.

It was only a few days ago that Google had indexed all of the pages (now over 100, since I've been adding new content).

My new site is in one of the most competitive niches, though, so I expect it will take longer for Google to index all of the pages and get the site ranked. I'm preparing myself for the possibility that it may take a year before the site starts to rank well.

If anyone thinks I'm wrong about this, please reply.

Meanwhile, I'm just getting inbound links from directories. Finding PR5+ sites in this very competitive niche for exchanging links is impossible. Every site owner who's ranking well knows the value of links from his site.


3:29 am on Sep 13, 2007 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

When it comes to backlinks, I don't concern myself very much with PR - just if the site looks of decent quality and not up to anything tricky. If I would like some of their exit traffic, then a link is good by me no matter what the toolbar says. I can see some links that are really cooking now, but they were new sites with PR0 when the link first showed up.


3:52 am on Sep 13, 2007 (gmt 0)

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member

When you are starting a new site, don't look for search engine power in your new links - look for targetted eyeballs. You want to get exposure to the people who might want to link to your site naturally; even if you need to pay for advertising units to get them to notice you.

Featured Threads

Hot Threads This Week

Hot Threads This Month