Google indexing pages in near real time

Something really cool is going on.

[JoeHouse]

Something really cool is going on and I am loving it!

Regarding indexing and BD. It appears Google either BD really likes my website or they have changed the way they are indexing sites....almost appears as "real time indexing"

Example:

In the morning I will check how many pages I have indexed with google. Then I add a bit of content and a couple hours later I check again and Google has indexed more of my pages. Then same day I add more content and I check again in a couple hours and "WOW" google has index more of my pages. So on and so on etc.....my pages indexed are continually climbing on an hourly basis.

I have never seen this behavior before with google.

This has been going on for several days but today its going through the roof!

Is anybody else seeing these "real time" indexing affects?

[Tearabite]

Big-Daddy G likes you more than me! :-(

I added some pages last night (16 hours ago) and they still aren't indexed..

.

[Lord Majestic]

Perhaps its just new index segments are being added to overall index - this explains growth in number of matches.

[g1smd]

I have seen new content appearing in the index about 16 to 30 hours after spidering, for almost two years now. If your site is spidered daily then your content appears very quickly.

Now if only Google could be a little less shy about throwing away cached pages for stuff that hasn't existed for two years.....

[Just Guessing]

At the moment I see my new pages being added to the index on non-BD datacenters within a couple of days of being crawled.

But the same pages are taking 2 weeks or longer to appear in the index on BD datacenters.

[BamaStangGuy]

The same thing is happening for me as well. I will check a hour or two in between each checks and they will have more pages indexed.

[annej]

Now if only Google could be a little less shy about throwing away cached pages for stuff that hasn't existed for two years

Boy do I agree with that! I keep checking every week or so to see if BD has dropped my nonexistant pages and they are always still there in the supplimentals.

[tedster]

The Google Sitemaps team indicated that supplemental results can hang around for years, and that they are not an issue we can influence, or that we should even be concerned about. I would only get into action mode if I saw a supplemental url that I know should be resolving, and there also was no url in the main index that resolves to that same content.

[g1smd]

The worry with many of the various supplemental results that I am trying to get Google to forget is that they are for out of date information: email addresses and phone numbers that we do not want exposed on the web any more, or which have changed, prices that are wrong, information that was incorrect and has been amended, and so on.

I don't mind archive.org holding on to that stuff because each page there is tagged with the crawl date, and there is an easy link to all of the other archived versions of each page, and each one has a date tag on it. Not so with Google, you get no big clue that you are looking at old information. Most surfers have no idea what a "Supplemental Result" is anyway; and if the page does not resolve, they may well look at the cache instead. That cache does have a date tag, but I guess that most people do not notice that, or think about it.

I am thinking of embedding a large date-time stamp on every page from now on, added as server side text: This information is only valid on: 2006 March 23rd (Javascript code would show a surfer the current date every time; I need a fixed date, fixed at the time of the crawl).

[traffik daddy]

I have 2 ecommerce sites selling the same products, both designed and optimised differently. One has been around longer than the other, say about 3 years old. The older of the two sites is better optimised in terms of inbound links, anchor text, etc as I have had more time to work on it than the other.

Over the last 2/3 weeks I have been adding pages to the new(ish) site, the pages still haven't been picked up yet, but within a day of importing the database to the older site the pages are picked up nearly instantly. I'm only importing the products and their prices so their is no dupe content as their is hardly any content at all. Within the database I have altered whatever little contnt their is. They are both the same PR but I'm page one for a very competitive keyword on Google with the older site.

One thing I have noticed though is that the pages are in and out of the index, especially over the last couple of days. One minute I have 800+ new pages indexed, then within hours they dissapear again.

My feeling that something big is on the horizon and when it happens I can't wait.

TD