Forum Moderators: open

Message Too Old, No Replies

is Googlebot intellegent?

do Google's bots determine their own navigation

         

10k_Guy

11:00 pm on Jan 2, 2003 (gmt 0)

10+ Year Member



Our ban has been lifted and we are waiting to be crawled. Our home page navigates to 5 countries, next level down is about 50 states and ultimately about 900 locations.

Each has a distinct URL and until we are crawled and indexed down to the location level we are not really back in business.

How do the bots navigate, does a postprocessing routine send the bot deeper next month or is the bot intellegent enough to determine its own navigation within my site on the first visit?

Hoople

6:01 am on Jan 3, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Mostly the bot finds you by links from other sites. It wouldn't hurt to submit the top level or some of the second level domains and see if that speeds the crawl. Most sites I have submitted were crawled within hours to a few days of submission. Some here will tell you submission does nothing. YMMV ;)

pageoneresults

6:11 am on Jan 3, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Based on what I've seen in the past and present, Google will typically go for your index page first, then follow links from there barring any robots.txt directives.

If you've mapped out your navigation and have created a roundtrip linking structure, Googlebot should dig deep. Maybe not on its first visit, but subsequent visits are most likely to occur at which time Googlebot will dig deeper.

P.S. Google's Freshbot seems to have a high level of intelligence too! ;)