| 3:09 am on Jul 15, 2008 (gmt 0)|
WMT seems to lose things. I don't often go there and was surprised to see some of my sites were unverified. I thought it was because G has renamed the verification file to uppercase while I had the old lowercase filename. Renamed the file and WMT is happy.
| 12:17 pm on Jul 15, 2008 (gmt 0)|
My site is still verified. Googlebot must not be reading the robots.txt file because there are urls listed that have the path through the cgi-bin folder. My ecommerce software uses JS to start it and injects a adding pathway through the cgi-bin - for example. If a user would go to a product page /123.html and the ecommerce software wasn't already in the path, it would refresh the page and add /cgi-local/softcart.exe/123.html?E+scstore.
What I find in the unreachable urls section are 2 pathways for the same page :/cgi-local/softcart.exe/123.html?E+scstore and /123.html
I thought Googlebot doesn't execute JS. In years past, I have added Disallow: /cgi-local/ to my robots.txt file which solved that issue.
I have other ecommerce sites running with the same ecommerce software without a hitch.
Is Googlebot ignoring robots.txt and then considering this duplicate content?
Should I remove the /cgi-local/softcart.exe urls that are listed in WMT?
How can I further test whether googlebot is really having a problem reading my robots.txt file or its some other problem?
| 8:46 pm on Jul 15, 2008 (gmt 0)|
If you're seeing urls in WMT that are blocked by robots.txt, you might try a url removal request, based on robots.txt and see what results you get from that.
| 9:37 pm on Jul 15, 2008 (gmt 0)|
That's a good idea. I'll try that. But that still doesn't explain why googlebot's ignoring the robots.txt file. If I'm violating some Google guideline I'm unaware of would WMT still give me the same nebulous response "robots.txt file unreachable"?
| 9:54 pm on Jul 15, 2008 (gmt 0)|
Have you tried the robots.txt tool that Google offers within your WMT account? That may give you some clues.
|My ecommerce software uses JS to start it and injects a adding pathway through the cgi-bin |
| 11:52 am on Jul 16, 2008 (gmt 0)|
I guess the next steps is to figure out whether there are unrelated (to the robots.tx file) possible problems that would trigger this error.