Forum Moderators: Robert Charlton & goodroi
I recently launched my first site. After watching over this and other forums as well as 'optimization' tips, I believe I have everything laid out properly - keywords in page title and H1/H2 tags, meta descriptions, sitemap page, you name it. I would not consider my pages to be 'over-optimized' either - I'm not keyword stuffing, just following the best practices that are commonly suggested.
The only thing out of the ordinary is that that I am building my site with regional subdomains which contain the bulk of my content, such as "northeast.widgets.com". The main page (www.widgets.com) is simply one page with intro text and links to the subdomains available.
About three weeks ago I added my subdomain site to Google. I currently have about 65 pages on this subdomain. Within a week Google had picked up a handful of pages (4 or 5). At the suggestion of many on this board, I created a sitemap.xml.gz file for my site and added it to Google's webmaster tools. Late last week Google had indexed only 15 of 65 pages, despite them having the sitemap file (which Google had parsed and found no errors). This hadn't changed up through yesterday - I still had only 15 of 65 pages indexed.
This morning I checked again and it seems that I am now down to only 9 indexed pages! My site hasn't ever been fully indexed yet Google is dropping pages on me. The ONLY thing I've done in the past week which I can think may have impacted anything is that a few days ago I deleted the unique google html file that webmaster tools asks you to add for authentication purposes. I thought it was a one-time verification thing and didn't realize it had to stay there permanently. Yesterday I noticed my site in webmaster tools was back at 'unverified' so I put the file back. I don't see how that would trigger a removal of links from the search engine. One other thing I did (which I thought may have been what was causing Google to only index 15 pages) was that webmaster tools found 4 missing/404 pages - I fixed those two days ago, but it doesn't seem Google has attempted to rescan those links yet.
For what it's worth, the only 'crawl' date I can find is for those 404's which indicate August 31. So as best I can tell, my site hasn't been indexed since the 31st yet Google decided to drop some pages since the last crawl.
So, my question for the hive mind is:
- Why would Google drop pages from its engine, even when not all links had been indexed?
- Does anything in my website structure, or the changes I mention being made, have any bearing on what has happened?
- Is there any kind of fatal flaw in my setup? (subdomains containing majority of the content, etc)
- Any additional suggestions on getting all pages into the Google database?
Thanks in advance for your help - this board is a great resource for us 'unseasoned' folk! :-)
Get an average (PR5+) or strong back-links (PR7+) from older sites linking to your subdomain and you will be get fully indexed within couple of days (depending on the competitiveness of XYZ pages).
My observations are that for non-competitive, non-commercial phrases (lets say around 1,5 mio - 2 mio of total results) couple of back-links from PR6+ long time ago established websites will make my targeted pages into top 5 SERPs usually within 3 days. PR8+ back-link made it within 10 hours...
From a technical point of view all your other steps taken seems to be OK. Due to the lack of backlinks, your subdomain seems to be unimportant to spiders and there they do not bother to index it.
in my view it is the lack of incoming links to your new subdomain
Highly likely that this is the cause. Sites with lots of links, especially those of high value, get attention and are indexed quickly.
Unfortunately, quality backlinks just don’t grow on trees.
I'll definitely look into this though.. I was just hoping my site could (at the very least) get its foot 'in the door' on its own merits rather than having to beg one of the 'big guys' to link to me just so Google will even bother to look at me.
As for 'dropping' pages, that's just flux. They'll be back in a few days.
Try to get a few links, but I would strongly advise you stay away from the paid route, or any spammy directories. That will likely just open worse problems.
Go do something else, and let it cook. Put a little time into getting links, but don't sweat it.
You're right, just sitting there and getting a few backlinks from scraper sites can be tough start-up. However, at some point early on, Google often gives new sites a boost in the SERPs for a day or two - I've called it the honeymoon in the past. That might help spread the word, too.
It was only a few days ago that Google had indexed all of the pages (now over 100, since I've been adding new content).
My new site is in one of the most competitive niches, though, so I expect it will take longer for Google to index all of the pages and get the site ranked. I'm preparing myself for the possibility that it may take a year before the site starts to rank well.
If anyone thinks I'm wrong about this, please reply.
Meanwhile, I'm just getting inbound links from directories. Finding PR5+ sites in this very competitive niche for exchanging links is impossible. Every site owner who's ranking well knows the value of links from his site.
