Only crawling a few pages

[dfunk]

Googlebot comes to our site quite frequently. But seems to be only crawling the same pages over and over again. Of about 400+ pages only 8 or 9 get crawled daily. If you do a search inurl:page_url site:domain.com most of the pages are in there.

Google sitemap seems to have no effect.

We also seem to be out of google completely since May 16th 2005. Which I'm guessing is the bourbon update. We've made all pages duplicate urls do a 301 to our main url. Upon dropping from google our other top 2 compeitors were dropped as well.

Thoughts on this?

[Brett_Tabke]

A lethargic GoogleBot is a symptom of too few links and/or low quality inbound links.

Go for more links and more deep links from other sites. Also do as much insite inner linking with yourself as possible. Make sure Gbot can crawl the entire thing. (keep everything to 2 clicks from root...)

[oddsod]

>> A lethargic GoogleBot is a symptom of too few links and/or low quality inbound links.

Brett, is there anything else that could be causing it? I've got a site that just can't be described by the above comment but GoogleBot is taking less than 30 pages a day (right up to a few months ago it was crawling a lot of pages. The site itself has several PR6 pages derived from thousands of IBLs... many from very reputed sites)

Does the www/non-www duplicate issue or something else possibly have any effect on the crawling frequency?

[stinkfoot]

In the same boat. 400 pages a few only crawled, sometimes none .. sitemaps set up and sending daily. incoming links 250 on google 9,040 on msn .. well established site (5 years or so) whats going on .. any ideas appreciated having contacted google many times no real responce.

[Brett_Tabke]

Offhand, if you have pr6, that should be more than enough to trigger some serious crawling. I'd bet in your case Oddsod, that it is an update frequency issue. You look static - your pages say they are static - and your last modified date is always the same. Update some pages a bit and reset some internal links. Feed that pr6 down into deep content as much as possible.

[oddsod]

Thanks, BT, you've obviously had a look at the site ;). I'll get to work on some of that. I appreciate your input on this.

[stinkfoot]

Bah .. googlebot need a good kick up the bum imo ... stupid robot. Unique content which they say they want) good structure (definite levels on information) good incoming and internal links, not tricks just good old fashioned regularly updated html, what more is to be done? grrrrr

I know ill change the colour scheme to a garish green royal blue and crimson red... thats bound to catch its eye

/kicks the dirt and dissapreas into the cloud of dust while ranting to himself again