g1smd
msg:4496328 | 5:06 pm on Sep 17, 2012 (gmt 0) |
That request is being redirected.
You should check where to. That could be an even bigger problem.
|
triggerfinger
msg:4496337 | 5:21 pm on Sep 17, 2012 (gmt 0) |
It gets redirected to the products page. Hence why we get errors, but why would google request a bogus URL like this?
|
triggerfinger
msg:4496362 | 6:42 pm on Sep 17, 2012 (gmt 0) |
Another clue: All the URLs seem to have vanity tld URLs redirecting to them. Is G trying to access the robots.txt of these URLs and instead requesting it from the deep page? Seems like a rather dumb idea for such a smart algorithm.
examplevanityurl.com -> 301 -> example.com/deepURL.html
examplevanityurl.com/robots.txt -> example.com/deepURL.html/robots.txt
Seems silly, no?
|
phranque
msg:4496391 | 7:57 pm on Sep 17, 2012 (gmt 0) |
if examplevanityurl.com is yours i would look for why that server is doing essentially a sitewide redirect to a subdirectory of example.com and fix it so it redirects to the root specifically for robots.txt request.
|
|
