Google Came and Left. Why?

Question about google robot behaviour

[scottj]

Dors anyone know why google would come to my
site, grab the top page. Wait 20 seconds, grab
the top page again and then stop.

There are 500,000 pages contained in
the site, full navigation links. Easy
for a robot to find all pages.

Are they just checking to see if the
site is real and they will come back
later to crawl?

[edited by: scottj at 3:51 am (utc) on Mar. 23, 2003]

[Stefan]

Welcome to Webmasterworld, Scottj.

Are they just checking to see if the
site is real and they will come back
later to crawl?

Probably the case, yes.

That's a very impressive site but you might want to remove the URL from the post, (TOS and all of that...)

<edit>edited for clarity</edit>

[Stefan]

Scott, is that site new? It seems as though it must have been in existence for some time. Has it never been listed in Google before?

[scottj]

Stefan,
The site is new. It has
never been listed before. But, there
is a subscription service which lets
you see more at a seperate site
(seperate dns name) -- and that
site has existed for a long time --
but that site does not alow robots
except at a few top-level pages.

Why? Will the content not be indexed
if the content appears elsewhere
even if unindexable?

Thanks.

-Scott

[Stefan]

I was asking more out of curiousity than anything else. It seemed that it couldn't have suddenly sprung up out of nowhere.

If the other site has been excluding the Google bots from most of the pages, and continues to do so, then there shouldn't be a chance of a duplicate content problem. I'd imagine you'll see Google back to get the rest of the new site's pages eventually, next deepcrawl anyway.