Forum Moderators: open

Message Too Old, No Replies

Google Deepcrawl

What path does Googlebot take?

         

Birdman

7:37 pm on Jan 24, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Let's suppose I change the whole structure of a site. When googlebot comes for the deepcrawl, does it:
  • start at the / page and start following links?
or
  • Start with a list of your already indexed pages and go off for them, regardless if the link still exists on the home page?

Thanks

Brett_Tabke

4:38 pm on Jan 27, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Hard to say where it is going to start. Given the nature of fresh bot and link checking, it could start anywhere. It's probably going to respider everything it already has, then come back for the new links it runs into a little later.

The path and method on that, is it will usually walk back to root and then start down into the site from there.

PaulPaul

4:41 pm on Jan 27, 2003 (gmt 0)

10+ Year Member



From my experience, I have found that more often than not, the second method is what googlebot does..

stuntdubl

6:41 pm on Jan 27, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Does this explain why sometime the number of "site: inurl" pages go down?

I have wondered how this can happend, is it relative to this question? I.E. If the whole site does not get re-spidered, some of the pages will get left out, even though they were in prior indexes.

If this is the case....
how can one make sure that the spider crawls the maximum number of pages? Will smaller file sizes on the pages help in this respect?

Just call me the "questionator":)