A Question to indexing

how often does Google really index?

[Taghor]

Hello all,

the situation at our website is the following:
We had a big update of our website in OCT 2002.
In this update we made the whole site managed by a own Website-Managing System written in PHP.
The URLs looked like this:
index.php3?page=public/izh/....

At the google dance at the end of OCT we were screwed :(
Google only did index the Index site and never followed any links. So we only have one result in the Google search.

We then started a research (in this forum too :)) and figured out that the dynamic links seem to be problem. So i changed yesterday all URLs with the rewrité mod of Apache.

The statistic today shows that Google was at our site in the night and visited all documents.

My question now is, when do the results of this visits show up in the database of goole?
really only at the next google-dance? Then why does Google crawl our whole site?

Thx for your answers in advance

Taghor

[johnser]

Taghor
If you have had many of your pages crawled by googlebot, and assuming some inbound links to your site, then you should expect to see results in G at the end of November.

The crawl is a monthly thing & follows whats called the Google "Dance". Do a site search (on WW) for "Google Dance" & you'll get a lot of useful answers.
J

[Taghor]

Sure, the problem is not being listed in Google we were there before and are now (but only wiht 1 document)

But i ask myself why does he crawl my whole site if he doesn't update his index before the great google dance?

Taghor

[johnser]

They have 3 billion + pages indexed. I've no idea but I'm guessing it takes time for their system to operate consistently with that much data!

[jdMorgan]

Taghor,

Welcome to WebmasterWorld!

Google also has another robot which may help you. This is the "freshbot" which often visits sites, but does not crawl as deeply as the monthly crawls.

If the freshbot picks up some of your pages, they may re-appear in the index. However, they may also disappear and reappear periodically until your site gets another deep crawl.

Check out the WebmasterWorld Library [webmasterworld.com] and try the WebmasterWorld Site Search [searchengineworld.com] for much more info!

Jim

[Taghor]

Thx

so its hope that we won't loose a whole month. :)

How long does it take after the freshbot visted me? any odeas about that? or is it just random?

And how can i find out that it was the freshbot who crawled me?

Thx for all answers

Taghor

[taxpod]

You can tell by the IP address of the bot. But if you've been visited in the past three days and had all your pages pulled down, I think it's fair to say that is likely the update bot. For that you will have to wait until the next dance. (The IP addresses are listed in a thread titled something like googlebot out walking.)

The freshbot is sort of random as to when you might get some more pages in. The important thing is that you seem to have fixed the problem. Maybe the Google Gods will be kind to you and you'll get a bunch of important pages in before the next dance.

[Taghor]

I searched this forum and found info that Bots form the Ip Range:
64.68.82.* are most likely the freshbot.

Is this confirmed or only a guess?

Caus the bot which crawled us was from 64.68.82.39.

Do you know how long it takes after the freshbot visited you until it shows the dociments in the search of google?

Sorry if i sound impatient. But i am very interested in that theme and we lost much Sessions on our website cause of the mistake with the dynamic sites :(

So i try to get it in the search again as soon as possible.

Thx again for all the answers.

Taghor

[caddie]

Things you should know

Freshbot ip address starts with 64.

Main crawler ip address starts with 216.

Freshbot crawls pages with pr4 and above regularly and normally upodates on the serps in around 2 to 3 days.

Hope this helps.