Going nuts with this one. I have MYSQL database feed site with pages that return products with links like this at the bottom:
The server log is showing Googlebot getting links like /category/30/30/widget.html or /category/1621/1650/widgets.html. Google is getting lots of invalid URLs and getting 200's in the log for each one.
The problem is that I can't replicate this in the browser or even using Lynx on my Suse machine. It's a fairly new site and Yahoo and MSN aren't spidering it so I can't compare there.
tedster: Sorry I didn't see your response since I don't get email notification even when I check it.
What I am seeing is /category/30/30/widget.html in the server log for the Googlebot with a 200 code. This URL does not exist in my code or can I make it produce it.
I may have solved the problem with a more educated pagination after digging deeper into MySQL. My links are now like 1¦2¦3¦ showing page numbers like Google rather than result ranges as before. Larry Ullman's book clarified some things for me now to wait and see how G likes it.
"What I am seeing is /category/30/30/widget.html in the server log for the Googlebot with a 200 code. This URL does not exist in my code or can I make it produce it."
You probably have another issue as well, if those pages are returning a 200 then your server is probably returning a _custom error page_ for pages that don't exist without returning the proper header response.
That _could_ be (and likely is) a major problem for your site.