Forum Moderators: Robert Charlton & goodroi
I have a couple of maintenance questions regarding site maps and the Google Webmaster tools. Last week we re-released a section of our website to make it more google friendly (created a proper directory structure, gave pages and directories meaningful names and removed the frames from the pages).
The result of this was 1,300 odd pages being redirected from /AllOurWidgets/... to /keywordwidgets/.../.../
On a functional level this has worked really well, anyone who looks up any of the old pages gets redirected to the new location, however Google webmaster tools is having a little trouble.
Initally there was a robots block on the /AllOurWidgets/ directory when the new area went live. This was mistake #1 as google would not be able to follow the old pages through the redirect to the new ones. The robots block was only up for 3 days, however it was too late, Google had already read it. G WMT is now reporting 1,032 URL's restricted by robots 7 days later. Just wondering how long it will take google to stop reporting the warning?
The second related problem is we now have 279 "Errors for URLs in Sitemaps" for those same /AllOurWidgets/ pages that now googles sees as not being linked from anywhere (as the parent pages are blocked). Will this clear itself up once Google realizes there is no restriction in robots anymore?
The third warning we have is to do with the 301's. There are a group of URL's that return the following message:
"When we tested a sample of the URLs from your Sitemap, we found that some URLs were not accessible to Googlebot because they contained too many redirects. Please change the URLs in your Sitemap that redirect and replace them with the destination URL (the redirect target). All valid URLs will still be submitted."
I am guessing the solution to this problem is to run a google sitemap tool over the site again, but do I ask for the tool to ignore the old /AllOurWidgets/ directory (keeping in mind that the old pages are still there, just everything redirects to the new pages) or allow it to add them to the sitemap? I have just done a run of the google sitemap(ing) tool and it seems to not follow 301's which is a little weird as 301's are a server side commands that should be processed before the tool picks it up.
Any suggestions would be a great help.
Thanks in advance ^_^
When the client receives a 301-redirect response from a server, it *may* elect to re-fetch the resource using the new URL. That is to say that the client usually *will* re-fetch the resource, but that this is the client's decision to make. If it does re-fetch the resource using the new URL, then this is an entirely new HTTP client-request/server-response transaction.
Remove the old URLs from your SiteMap(s), redirect the old URLs to the new, and remove all robots.txt restrictions on all old and new URLs -- You must allow the 'bot to fetch the old URLs in order for it to see the redirects.
Having done that, it may take a few weeks for this to clear up; In the meantime, I suggest you adopt a "hands off" policy with regard to making any additional changes or additions with respect to the old or new URLs, so as not to confuse the 'bot.
Jim
I will do a excluded run of the sitemap - then leave it sit.
You need to make sure that there is just one step from old to new URL.
If you have "chained" redirects, that will cause you problems. That occurs when the redirect rules are in the wrong order in your .htaccess file.
You need to ask for A and be taken directly to B.
If asking for A takes you to another URL and then another URL and then finally to B, they you do have a problem that needs to be fixed.
