How does google crawl

[Posebud]

I have a website that has majority of pages indexed as mysite.com/page1 and so on. I created several new pages and google crawled these as mysite.com/mysite/page1. I have no links to this url so PR is non existent. I'm not sure if this is a yahoo store problem or a google problem. I can't find any links out there that would make google crawl my site like this.

The only thing I can figure out is, if you visit mysite.com/nonexistentpage.html yahoo store redirects you to mysite.com/mysite/. Even if google came to this page, shouldn't it follow the links on the page which are correct?

Any help would be greatly appreciated

Thanks,

Greg

[pmac]

Hi Greg, and welcome to WebmasterWorld.

I'm not sure if I understand the question, but Google will not crawl a url that does not have some kind of link pointing to it.

Make sure that your internal site navigation links to the pages that you want indexed. A site map is a simple way of making sure that Google can find your pages.

[Posebud]

Hi,

Thanks for the response. Our internal links all point correctly, and other search engines are fine. For instance though, all of our pages link back to the home page of www.mydomain.com. However, google is crawling some of our pages like www.mydomain.com/mydomain/. This link still works and takes you to our site, but all PR is lost from internal linking

[edited by: ciml at 4:16 pm (utc) on Feb. 7, 2003]
[edit reason] No specifics please. [/edit]