GoogleBot Crawls my site like a Spider with 2 Legs

[HorsepowerFreak]

I have over 5,000 pages of pure content with all rules "that I know of" of proper HTML employed. On Yahoo, I get 1,100 referrals/day. I have requested that Google crawl the site many times. I checked the status of the spider using "Robot Manager" and found that Googlebot is crawling 1 page/day, and these pages are no longer referred to, but I left them around just in case.

Why is Googlebot so slow? Some of the other ones will crawl all 5,000 pages in one day. I will never get my site listed at this rate. Is there anything I can do?

Chris.

[Brett_Tabke]

Higher pr cures all. Get more inbound links. A slow gbot is indicative of low pr.

[div01]

You also have to make sure that you have search engine friendly links.

[HorsepowerFreak]

This is going to sound very very stupid and a newbie question. But how to I check the PR of the site from google? Also, am I allowed to post the name of my site here? I see nobody else does that.

[Macro]

Download the Google toolbar and all will be revealed: [toolbar.google.com...]

[jimh009]

Google does not like frames that much. If the site in questioin is the one used in your email address in your profile, that may explain it. I also found the navigation on the site rather difficult. If I found it difficult, Googlebot may have found it that way, too.

[HorsepowerFreak]

The site that refers people is a site that links to my main site. This one when you click on parts it drives people into my main site. That does not have frames and I tried to make it easy to crawl. If you can find issues with that, please let me know, I'd like to fix them.

P.S. I will download the google toolbar. Thanks.

Chris.

[edited by: WebGuerrilla at 10:11 pm (utc) on Sep. 28, 2004]
[edit reason] removed specifics [/edit]

[HorsepowerFreak]

Well, i checked the page rank. My referring site rank (my domain name without the "s" on the end) is I assume terrible. 1/10. My main site rank is probably pretty crappy as well, 3/10. My main site has thousands of remote links to it but it uses frames to make it easy for the customer to browse our product line. My referring site probably doesn't have very many links to it and is not very easy to browse our product line but has tons of content.

I assume I need to try to do some things to improve the PR.

Any easy ways to do it or do I need to somehow have lots of web-sites that link to mine. It kind of sucks because Yahoo and MSN treat my referring site like a king and give me 2,000 referrals/day combined. Google gives me hardly any.

Chris.

[funandgames]

I have heard from many sources that the Google Toolbar isnt updating the PR anymore, or that it is three months old data.

[DerekH]

Horsepowerfreak - I'll give you the same advice that I seem to keep giving, though I don't mean that to sound negative! - run your site through the W3C validator and then correct the 40 or so errors on your home page. You say it's good HTML, but the W3C validator is adamant it's not that good at all!
Google has a good page of advice at
h-tea-tea-pea://www.google.com/webmasters/guidelines.html

It's always worth getting rid of HTML errors - we're never quite sure whether spiders, in their rush round the net, are quite as forgiving as browsers, when you make coding errors. HTML errors that cause the spider to jump over a link will mean that link is missed.
Regards
DerekH

[HorsepowerFreak]

I tried that tool. Awesome information. I will try it and fix the errors.

Thanks.
Chris.

[HorsepowerFreak]

I reworked all 5,000 pages to be W3C compliant. Do you see anything else wrong with the site?

Thanks.
Chris.

[Powdork]

Damn, that was quick.:)

A hint about frames. Your PR is transferred from the page (read frame here) that has the link on it. To figure this, right click on the frame, view properties, paste the frames url into your browser's address bar and then view the page's rank. But yes, the PR given by the toolbar should only be used in a medical emergency.;)
Doing this you will see that your navigation frame on the domain with the s has a PR of zero (according to the toolbar).