Googlebot/2.1 and Pagination

[bumpaw]

Going nuts with this one. I have MYSQL database feed site with pages that return products with links like this at the bottom:

[3]¦1-30¦31-60¦61-90¦Next¦[/3]

The server log is showing Googlebot getting links like /category/30/30/widget.html or /category/1621/1650/widgets.html. Google is getting lots of invalid URLs and getting 200's in the log for each one.

The problem is that I can't replicate this in the browser or even using Lynx on my Suse machine. It's a fairly new site and Yahoo and MSN aren't spidering it so I can't compare there.

[tedster]

I can't replicate this in the browser

Do you mean the server is returning a 404 to your browser, but the logs say it returns 200 to googlebot?

[bumpaw]

tedster: Sorry I didn't see your response since I don't get email notification even when I check it.

What I am seeing is /category/30/30/widget.html in the server log for the Googlebot with a 200 code. This URL does not exist in my code or can I make it produce it.

I may have solved the problem with a more educated pagination after digging deeper into MySQL. My links are now like 1¦2¦3¦ showing page numbers like Google rather than result ranges as before. Larry Ullman's book clarified some things for me now to wait and see how G likes it.

[theBear]

"What I am seeing is /category/30/30/widget.html in the server log for the Googlebot with a 200 code. This URL does not exist in my code or can I make it produce it."

You probably have another issue as well, if those pages are returning a 200 then your server is probably returning a _custom error page_ for pages that don't exist without returning the proper header response.

That _could_ be (and likely is) a major problem for your site.

[bumpaw]

"GET /russ/cat2/30/30/gzhel.html HTTP/1.1" 200 38657 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

It looks like the bot loaded 38657. Are you thinking that could be the custom error page?

[theBear]

It sure looks like it could be one, now the question would be where does it come from your content management system or via the server configuration.