homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 60 message thread spans 2 pages: 60 ( [1] 2 > >     
Lag between crawl and index appearance in Google
what's the usual time period nowadays

 1:21 am on May 21, 2004 (gmt 0)


Pls be patient with the newbie :-)

Google has sent both its MediaPartners and the regular Googlebot to a new site of mine, and they have extensively crawled the site. Furthermore, I can see how every day they are coming back to the site, spidering away.

The site went live three weeks ago. The first intense Googlebot session came about 1 day after go-live. But although it is now abundantly clear that Google's bots know about the domain, and regular visitors, today when you do a Google search on 'site: aabbccdd.com', Google says: "Your search - site:aabbccdd.com - did not match any documents".

Furthermore, when you do a Google search for the domain, simply as: "aabbccdd.com" Google says:

" Sorry, no information is available for the URL aabbccdd.com"

And no results appear from the site in any search results when you do even the most detailed keyword search.

Given the impressive speed with which the Googlebots first found the site and came around and begun indexing it three weeks ago, should I expect to see the site in Google yet?

Or is it too early, and should I wait a bit longer before assuming something is amiss.

I don't feel as if there is anything on the site that should penalise it or get it blocked. It is a G-rated vanilla standard content rich site with no weird or exotic SEO attempted, and with a valid robots.txt.

The only 'SEO' attempted, aside from good titling, metadata and the rest of it, are just 3 external links to it off 2 other sites I run, from a different host. I have also got all the gTLD's for the same site, and have them resolving to the one domain too, but that is only aabbccdd.com/net/org/info. So nothing extreme here either.

Should I just be a bit more patient? I am aware that in the old days you had to wait forever to get into people like AltaVista's index, but I am assuming nowadays getting into Google's index is a lot quicker.



 11:30 am on May 21, 2004 (gmt 0)

After three weeks it should be in the index, but sometimes it takes 6 weeks or even more.

You are likey not penalise and still on the long list of sites waiting to be in the index.


 2:33 pm on May 21, 2004 (gmt 0)

This is a good topic, and one that I've heard a lot of recently.

Back in the day, Google updated almost on a monthly cycle. Nowadays, it is more of a continuous update cycle that few can predict.

The only way to get a site into Google faster is to work on getting relevant inbound links. To help that process, I would recommend getting links from pages that often have fresh tags, since you know Google is at least coming by often. Also, do not always focus your efforts on the domain name itself... Be sure to sprinkle some inbound links to deeper content pages.

Sad to say, but much of the initial Google optimization efforts involves waiting.


 5:06 pm on May 21, 2004 (gmt 0)

Thanks guys.

I think I'll wait it out and see what happens over the coming weeks.

I am a firm believer in not over-relying on search engines for traffic, so the absence of a Google listing to date, and any search engine traffic, is a good simulation of what life would be like without the SE's.

The delay in listing encourages me to put more effort into other marketing, which is probably a good thing.

Although it is hard nowadays, one should probably design and operate a site so that it could survive without SE traffic anyhow. In theory. Perhaps. Maybe.

In the past (before Google's hegemony), I've run high-traffic sites that although they ranked well in the SE, only drew 30% of their traffic from SE referrals, bringing in the rest through good link programs and also lots of simple marketing and word-of-mouth.

That seems a good formula to try and repeat.


 2:45 am on Jun 1, 2004 (gmt 0)

Just an update:

It is now exactly 1 month since the site went live, and it still isn't in the Google index. Not just one or two pages absent from the index, but the whole site of about 100 pages + a forum just doesn't exist there when you search for 'site: blah.com'

This is despite the fact that the Googlebots are crawling the site *constantly* now.

In fact, yesterday I added several new articles and was amazed to see the Googlebot come and crawl them just 5 minutes after I'd uploaded them.

But still no index inclusion. Weird.

It's a good thing patience is a virtue.


 10:00 am on Jun 1, 2004 (gmt 0)


There has been a Link Update, check if your site show 0 Backlinks?


 11:00 am on Jun 1, 2004 (gmt 0)

There are about the same number of backlinks as before. There are a few pages which mention the site, and link to my site's home page URL, but Goggle still doesn't show any results which are actually *from* my site, even though I know that the Googlebot has in fact visited my site and crawled its pages over 400 times in May.

I guess I wait a little bit more and see what happens.


 12:34 pm on Jun 1, 2004 (gmt 0)

Patience is the word for you Moocow. Every thing is going well. Just wait and watch.


 3:30 am on Jun 2, 2004 (gmt 0)

I completed my first freelance web site gig in January. I thought I did a decent job of paying attention to the search engines and to accessibility by using text as much as possible, using alt tags to describe images, writing code that validates, using h1/h2/etc, using file names that describe what the page is about, and using descriptive titles on pages.

I submitted to google (before I read here that you shouldn't) and waited. I knew it would take a long time. There was a listing in google for the site, but it still reflected the "under construction" notice that had been gone from the site for a month. It also was listed as a "Supplemental Result".

I waited and watched. Nothing.

I understand that a site with zero inbound links is not going to get listed for hot keywords, but shouldn't I be able to expect at least an accurate result/description/title in google when I search for "exact site name here"? I mean, the site name, company name, and domain name are all the same. that should be a no brainer for google when i type that in, right? MSN, Yahoo, and several others have accurate descriptions and results. But google has me wondering what is up.

Is it possible my PR0 is a result of a penalty from google? I did use alt tags on every image. I've read that can be a problem. In fact, as I type, I realize I used "Artwork, click for description" as the alt text for 100-200 images. I can see how, theoretically, that could be viewed as spam to a search engine. But that seems ludicrous. There must be millions of people like me who want to write accessible pages, but don't have the time to customize every alt tag.

Also, the vast majority of the 200 pages on the site are from a template that I created for images, they are basically the same with different images and captions. Is this possibly regarded as spam by google? Or do i just need to convince the client that the only thing we can do is increase the number of inbound links?

I picked a bad time to learn how SEO works, at least as far as google goes. even googleguy is quiet! what's a newbie web designer to do?

Thanks for reading. I know i'm long winded. I have been lurking for months, I guess I had a lot of things to say all saved up.



 5:26 am on Jun 2, 2004 (gmt 0)

I just bought a domain on Friday, built 30 pages over the weekend put them up on Sunday, it got crawled cached/indexed today and is showing #321 of 97,000 results a few hours later. Only the index got crawled though, just one page, and we're waiting on the rest. I'll let you know how fast they get in.

I'd say a month is unusual if the site is getting deepcrawled.

Depends about the links to your site though maybe. We've got some PR3,4,5 and 6 links.


 2:01 pm on Jun 3, 2004 (gmt 0)

Here's an update to my plea for advice:

I added a link to the client's PR0 site from my (personal) PR2 site, and within 9 days the client's site was updated in google, the cached version is now of a relatively up to date version of the site. Is it possible this is just a coincidence?


 6:45 pm on Jun 3, 2004 (gmt 0)

It hasn't helped me yet.

I have linked to my (newly created site) from two of my own sites (on a different server) that have PR 5 and PR6.

Hasn't made one whit of difference. One month and counting, and still not in the G. index.

I have two other biggish content sites in the pipeline. I might make one of those active in skeleton form and see how quickly it is picked up.

If indeed sometimes it can only take a matter of days to get into the index, as one of the posters here mentioned, seeing how this site is treated by Google might help me narrow down the reasons for the delay in getting the other one properly grokked by Google.


 2:46 pm on Jun 12, 2004 (gmt 0)

Just another update - 13 June and still counting, which makes 6 weeks from the first time the Googlebot came and visited the site - no appearance of the site in the Google index, except now for a few *external* mentions of the site URL on a few subject related forums.

Meanwhile the Google spider is still visiting everyday, as (after a few weeks delay over the Googlebot) is the MSN robot (which is very thorough, Yahoo (ditto), Alexa etc etc.

At this rate, Yahoo will beat Google to the punch.

All in all, Google is something of a disappointment.

I never thought I'd see the day when the Inktomi/Yahoo index was likely to be fresher than Google's.

A sign of the times for Google, perhaps.


 2:57 pm on Jun 12, 2004 (gmt 0)

[quote[At this rate, Yahoo will beat Google to the punch.

All in all, Google is something of a disappointment.

I never thought I'd see the day when the Inktomi/Yahoo index was likely to be fresher than Google's.

A sign of the times for Google, perhaps.[/quote]

I was waiting too see how long it would take before we went down this road :)



 3:16 pm on Jun 12, 2004 (gmt 0)

I think I have been having a similar experience to moocow's. It hasn't been going on as long, but I have the frustrating feeling that Google is ignoring my site.

I am no web professional, but I've had occasion to help tell the world about a few websites. In the past, when I submitted a URL to Google, it appeared in the index after a couple of days at most.

Now I have a new site, on its own newly purchased .org domain, that I'd like people to be able to find using a search engine. I don't care about the page rank -- I just want it to be in the index for people who are looking for it specifically.

I submitted the URL to Google a couple of weeks ago. There are two publicly visible links to it: one from a PR6 site (although the new page linking to it is still just PR0) and one from a PR4 page on a site that is PR9. Although I assume some of my site visitors have done the same, I have visited the site from MSIE a couple of times (from different IPs) with the Google tracking feature on, in hopes of luring Googlebot.

Googlebot finally visited the site a few hours ago, looked only at the index page, and left.

I'm hoping Googlebot will be back, and that at least the site will show up in the index. However, I'm puzzled that it took "him" so long to get here, and I am wondering if there is some reason why he's been avoiding this site.

One concern I have is that the domain apparently was registered to someone else once (according to the data at a whois site), but before I registered it I found no shadows of the old site anywhere on the web (no links to it, no cached version on archive.org). Is Google possibly penalizing the domain?


 8:11 pm on Jun 12, 2004 (gmt 0)

Pre-owned domains might be a problem. Google doesn't make exisiting incoming links that they had already indexed, count for the new site under the new owner, after the change of ownership.

I would try to get some more links to your site, and hope that Google comes sniffing round soon.


I am aware of a new site that went online mid-May and was spidered by Google 8 hours later (it had one link to it on another site with a PR6 page that is indexed every day or two, and another on a PR7 page elsewhere). The site appeared in search results at #6 about 30 hours later, with a fresh tag dated the same as the upload day.

A few weeks later I returned to see that it had 2 inbound links listed for it in Google but was still showing a PR of zero. I then arranged for about 50 related sites (extracted by looking through the top 100 sites for related keyword searches, and sending a brief email) to provide one incoming link each. Many responed the same or next day, and added a link. A lot more content was also added, and the site jumped to #2 when the next fresh date appeared another 30 or so hours later.

When the incoming links are factored in, in the next few weeks, it might even get to #1. SERPs are 98000 results.

Page hits for 2004-06-08 were 100143. This isn't my own site, but is one where I helped the owner out by dropping off a few emails for links.


Site isn't in the ODP, but there is an editor note against the URL to say that I have been helping out with a few things.


 9:07 pm on Jun 12, 2004 (gmt 0)

Last night I uploaded a small local political campaign site:
1. robots.txt
2. site map
3. handful of links from PR3 and PR4 sites
4. Submitted URL to Google
5. Submitted to DMOZ
6. submitted to larger political directory sites

So let's see what happens. So far, no sign of google bots in the logs, (but an incredible amount of "junk" traffic!)

So let's belly up to bar: How long before we're live in Google?



 9:25 pm on Jun 12, 2004 (gmt 0)

You could be listed as soon as about 10 to 30 hours after the Googlebot first sniffs around.

How long before it arrives will depend on having some nice links into your site.


 9:46 pm on Jun 12, 2004 (gmt 0)

So far, five PR3 pages link to us.

One PR6 pending....


 11:20 pm on Jun 12, 2004 (gmt 0)

There are two publicly visible links to it: one from a PR6 site (although the new page linking to it is still just PR0) and one from a PR4 page on a site that is PR9.;

Site PR is much less important than the PR of the page linking to this new site. The index page of PR9 site will likely be crawled at least once a day but the PR4 page might not be crawled for weeks (or even months) and until that time Google might not know the existence of this new site.

I think that the best solution, if in a hurry, is to get a direct link from a PR6 or more page.


 4:06 am on Jun 13, 2004 (gmt 0)

Well, now when I enter the domain name in the Google search box, it shows a link to my site and displays some text from the home page. If I search Google on key words from the page, however, I don't get a hit on the page -- but I do get a hit on one of the pages that links to it (the PR0 page on the PR6 site, which is a daily newspaper that gets crawled daily).

It will be interesting to see what happens next. However, this slow discovery of the site was a surprise after my earlier experience.


 7:22 am on Jun 13, 2004 (gmt 0)

I had better results than that last year, but was still jittery about getting in: [webmasterworld.com...] - improve the links in, and maybe it will suddenly appear.


 4:48 pm on Jun 13, 2004 (gmt 0)

Googlebot has been slow coming to my site as well :(
I submitted it to Google's addurl four weeks ago, and haven't seen any sign of it. Albeit, I do not have any incoming links to my site, but I had hoped it would have at least stopped by. Is it a requirement now just to have incoming links for google to stop by?

(Surprisingly?) Yahoo and Ask Jeeves, on the other hand, have came to my site and done some spidering.


 7:17 pm on Jun 18, 2004 (gmt 0)

On June 12 I wrote:
"Last night I uploaded a small local political campaign site:
1. robots.txt
2. site map
3. handful of links from PR3 and PR4 sites
4. Submitted URL to Google
5. Submitted to DMOZ
6. submitted to larger political directory sites"

Yesterday the spiders came through en mass. First thing they hit was the site map.

No listing yet.


 7:57 pm on Jun 18, 2004 (gmt 0)

Perhaps I am missing something, but I think this thread is asking how long does a site wait in the "sandbox" before anyone can find it. Our experience is that the pages get crawled and will even show up in a site: query after a few days. But getting anywhere in the SERPs usually takes from 80 to 110 days. We have observed this pattern three times over the course of the last year.


 8:14 pm on Jun 18, 2004 (gmt 0)

Yay, sort of.

Today is the first time that I've seen results for site:www.mysite.com appearing in the Google index.

So the lag in getting in the index was from 1 May to today, about 7 weeks.

PR of the site at present is 0, which for a new site is to be expected, and it is also clear that although Google has spidered every page on site, only a selection are appearing in the index at the moment.

But that's progress. Now I'm happy.

Now that the pages are appearing though, it has also highlighted how there are a few things I need to clear up to make the Google search results more meaningful for people (forget about SEO - just people browsing the results).

For example my page titling is good and comprehensive, but too long, and gets truncated in the Google results display. I didn't think about how that would look to people. As the main subject of the page is at the end of the title line, it can be hard to pick up the subject of the page.

Etc. So the ball has started rolling on Google.

The Inktomi spider has also gone crazy over the site these last few weeks, so it will be interesting to see what the results are there.


 3:04 am on Jun 19, 2004 (gmt 0)

I'm going to use the new site as a "test bed" and follow this issue very carefully.


 3:09 am on Jun 19, 2004 (gmt 0)

Congrats, moocow!

Update on my situation:

Eleven visits by Googlebot now, including some to pages other than the home page. The site shows up on Google when I search on the domain name, but it does not appear in keyword-based search results and Google does not identify any sites as linking to it.

Interestingly, the server logs show that somebody somehow got to the site from att.net's websearch page, which is "powered by" Google (and does not list the site in search results).


 6:40 am on Jun 19, 2004 (gmt 0)

Well, I should qualify my previous 'yay' response.

While Google now returns 90+ site pages from its index when I do a site:www.mysite.com query, it is hard to get any of those pages to actually appear in any free-form Google queries yet.

I expect that will come soon. Hopefully.

I updated all the page titles a short while ago and it was surprising to see the Googlebot come around and suck up a big chunk of the site again shortly afterwards. At least the backend of Google seems pretty efficient.

Traffic from Yahoo/Altavista has started to flow consistently now from user search queries on those engines. Not hugely, but it is certainly there now.

The page descriptions being returned by Yahoo are quite good as well - the page's META Description text, followed by a snippet from the first para of the page. Classic stuff. Each page summary reeads quite well as a result.

It is still frustrating to see crappier and plainly less relevant pages turning up in the Yahoo results ahead of some of mine, but thems the breaks.

Having said that, it will be interesting to see what the pattern of search referrals is once more volume comes through, and (once Google starts returning results) it will also be interesting to compare the results for some key phrases coming back from both Yahoo and Google.

Fun fun fun.


 9:27 pm on Jun 19, 2004 (gmt 0)

I'm pleased to report that my site now turns up in the search results for free-form Google queries. :)

I hope that the data we presented here are helpful to others asking the same questions we have been asking, moocow.

This 60 message thread spans 2 pages: 60 ( [1] 2 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved