Forum Moderators: Robert Charlton & goodroi
Perhaps someone could help me? Here's the deal:
I have a widget site. On a search page you search by what location you need your widget. Then I deliver a results page with all of the widgets for that location. That results page has a link to widget detail page that shows the details of your widget.
For some reason Googlebot is crawling the results page, then instead of just following the links I provide to each resulting widget's detail page, it adds a "%5" to the link. Then in my WMT it tells me that these pages are not found.
But if you go to the site and check it (its pretty easy, theres only a few possible results) ALL the links to ALL the widget detail pages work. NONE of them have the "%5" in the link....never have.
How could this happening? And why the heck would Googlebot add something to a link?
Google is indexing site search results pages [webmasterworld.com]
Googlebot Now Crawls via HTML Forms [webmasterworld.com]
Oh Well, I was hoping to get some of the search result pages indexed as I felt they provided value to people looking for widgets in location ABC.
Anyway I guess I will have to block the search results pages until Goog straightens themselves out. Sometimes I think they are so awesome, and other times its like "dude you guys are the biggest and best SE and you can't train your bot to follow the links already presented to you in a results page....you decide to make up fake links?"
Thanks though Tedster you're always a great help.
This could be hidden in your HTML without you seeing it!
Like: if you copy text from a word doc ...
just a thing I would check before killing the idea of getting the results spidered!
P!