Forum Moderators: Robert Charlton & goodroi
The site has links pointing to it including links pointing to internal pages. The site itself has unique content.
Googlebot comes by and grabs pages each day but they are very slow being added to the index.
Can anyone who has launched a brand new site in the last month or so share their experience? How quickly has it been crawled and how quickly are you seeing pages added to the index?
My feeling is that during this BigDaddy transition, new pages may not be added at the same speed that they once were.
I also have a hunch that when pages are now crawled, they go through some rigorous scrutinity to determine if they are spam or quality pages.
Thanks.
My site went online Jan 20.
Google and MSN had my start page indexed within a couple of days..
MSN had 10 or so pages indexed a few days after that..
it took about 2 weeks for google to index about 20 of my pages, and another week after that for about 10 more.. currently non-big-daddy has about 32 of my pages indexed, and BD has 16.. MSN has 33 pages as of this morning..
Yahoo on the other hand STILL only has my start-page indexed.
A few questions. How many pages does your site have in total? Has Google spidered any of your pages but they are still to show in the index? Can you give us an idea of the links you have to the site (quantity, quality, where they point to). To you, does Google seem to be slower than usual at adding pages to the index for new sites?
Would be interested to hearing about the indexing of other new sites launched in the last month or two.
A site I launched mid last year was in the index within that time scale and instantly started to get targeted traffic, IE: main key phrases targeted were producing results, after a month it was no were to be seen but was still in the index and ONLY getting traffic for obscure phrases with low traffic.
For the record the site is STILL in the sandbox, whatever it is.
Forgot: it was instantly a PR4 when it was first in the index and still is (but that could change as we havent had a toolbar PR update since, one is due anytime soon, maybe after mid march D-Day for BD, they say.
It has been around 3 weeks now with just the homepage showing - even though G bot comes around nearly every day.
Strange thing is - whilst waiting for the new homepage to
show, i clicked on the cached link on the site:domain results, and the old content showed, but with a single image from the new page showing.
Ps....msn shows all 50 pages
burcot, that slow rate of indexing is pretty much what I am seeing. I am wondering if having to change (remove) the old content on the site you purchased put it back in the sandbox. A friend of mine did the same thing (bought and old site and removed the content) and it went back in the box.
Sorry to get off topic on my own thread, I am hoping to get more experiences on new sites being indexed by Google.
That being said, I'm not yet ranking too well for my main keywords. I'm guessing that's the ol' sandbox in effect and hopefully it will improve with time. Most of my referrals are coming from MSN. In my experience, that will remain the case for a few months, then suddenly I will rise in Google's SERPs and its referrals will quickly overtake MSN's by tenfold.
New domain, with all files uploaded to site the next day. Sitemap page listing all 200 pages placed on another existing site next day so that Google would easily find it. All 200 pages indexed in Google after 72 hours, and all still there. All show title and description, and cache is less than a week old on most of them.
PR: white bar on all pages apart from root index page, which shows "2".
What I currently suspect Google of doing is assigning a very high priority to completely new (i.e., new site) home pages. One,perhaps two, pages seem to appear in search results very quickly.
What I would like to try, but haven't had the opportunity yet, is to see if sitemaps PRIORITY has any effect at all. In practice, can the webmaster control which subset of pages the crawler goes to?
Bzuild a new site, put 100 pages up, add a sitemap with priority 1 for the index page (which it seems to get anyway) then a couple at 0.8, a couple at 0.6 and the rest at 0.0 - then see which ones get crawled first.
And don't anyone tell me that Google doesn't use sitemap priority unless you can prove it.
Another site I launched 3 weeks ago has not got 33 pages in non-BD, less in BD.
If you can get at least one PR7 link you're golden.
g1smd, I too launched a site in late Novemeber. It was a large site and it took about 3 weeks to get it full crawled and indexed. Interesting point you made with:
Sitemap page listing all 200 pages placed on another existing site next day so that Google would easily find it.
The fact that the sitemap was on another site may have really helped. These days, I am keeping my sites very separarate but that is something to keep in mind.
It looks as if crawling is running ahead of indexing and there's a fair wait to get a page indexed.
Phil, that is what I suspect too and it is nice to hear you might think so too. Interesting test you have planned in terms of priorities and site maps, would like to hear the results of that.
otech, the site you launched three weeks ago that has 33 pages in non-BD, how many pages does it have in total? Can you give us an idea of the links pointing to it?
I should add one thing. It seems links from other sites pointing to internal pages of my site have helped get those pages spidered. (Much like what g1smd has said, only I don't own or control those sites.) For me, there is some indication of this being effective for Google and definite indication of this working for MSN.
However I have also seen several sites that have been online for months with only the home page indexed. These were all written in ASP, PHP or another script, usually with session IDs. Make sure your site doesn't have session IDs or JavaScript menu unless you also provide a site map for the later.
of 50 pages about 30 are indexed. The links inbound were relevant to the genre I am in.
Hope this helps.
Really, I don't think there is a problem and that is confirmed by what comments from otech and Cain are relaying. Three weeks have passed and all their pages are not fully indexed. From my experience and memory, this was never the case with the Google of old.
As I remember things, you could put up a site. Get one link to it and be indexed in about three days (okay, maybe a week). This is not the case anymore, at least not usually from what we are hearing. Not sure if the increasing time to be indexed is caused by the BigDaddy move. I do think Google has gotten slower at indexing sites even from late last year.
Another theory to explain the slower speed of indexing (if that is a correct observation) is that Google has changed the way it crawls and indexes. Perhaps it needs more link power to go in and 'consume' a site and have those pages appear in the index more quickly. (This may always have been the case to some degree, seems much stronger now.) Perhaps it is doing some heavy duty one-time calculations on pages when a site is first crawled to see if they should be in the index at all?
Not sure what I think but if I had to guess, I would think this is something by design and intention and not just related to BigDaddy. If they are making sites wait in the sandbox, this could be just one more hurdle that sites need to jump.
Thanks all for sharing, this helps me understand that this is just one an isolated case I am experiencing.
Now, it usually requires on *good* (higher pr) link or a few lesser links.
So what you are saying makes total sense...
I'm now about 80% indexed by both Google and MSN, but I rank #3-5 on MSN for most of my keywords (and other keywords i never thought about)! Too bad nobody uses MSN search.. :-(
Yahoo still only has 1 page of mine indexed
Google has me ranked #5,000,000,000 (or there abouts) for most of my keywords.
Site has been online 30 days as of today.
If you can get at least one PR7 link you're golden.
Definitely agree with that, higher page rank links do help a site get noticed and indexed.
This is not necessarily true. If you have a brand new site and all of a sudden a PR7 site links to you, you'll end up in the sandbox for a long time. Generally the natural progression of incoming links shows lower PR sites linking to you, and as your site gains credibility the higher PR sites will also consider linking to you. So if all of a sudden a PR7 site links to you it would seem 'unnatural', especially if the PR7 is the only link you have.
