301 robots.txt file when domain move?

[jpeuron]

Hey!

I've helped a client moving domain recently from www.xyz.com to www.xyz2.com.

Every single URL on the old domain has been 301 redirected to their matching version on the new domain, including the robots.txt file that has been 301'd from www.xyz.com/robots.txt to www.xyz2.com/robots.txt

It's been a few weeks since the move now and Google is still returning some old URLs from the old domain in the result pages, and some pages on the old site even show that "A description of this result is not available because of the site's robots.txt"

Has anywone ever had a similar experience? Is the robots.txt not supposed to be 301 redirected when a site is moving domain?

Thank you!

Johan

[aakk9999]

Welcome to WebmasterWorld, jpeuron!

Google advises removing robots.txt on the old site:

Start the site move
https://support.google.com/webmasters/answer/6033080?hl=en [support.google.com]

On the source site, remove all robots.txt directives. This allows Googlebot to discover all redirects to the new site and update our index.

However if the site being moved is very large and has lots of blocked resources, I would personally not remove robots.txt as this could result in Google attempting to crawl many URLs that were previously blocked and waste its crawling budget in this way. Instead I would keep the original robots.txt without redirecting.

On the other hand, many webmasters do redirect robots.txt (especially when it is the same) and they have not seen any adverse effect during the site move.

Regarding Google still returning some old URLs despite being a few weeks since the move - this is normal and it may take more than a few weeks for Google to process all redirects. You do not say how big is the site, but with bigger sites it could take months. Have you checked your server logs to see whether Google has crawled old site URL in order to see the redirect?

What may help is - providing you have Google Search Console account for the old site - add the new site to the same Google Search Console account and then execute Change of Address. You can find more details here:

Use the change of address tool
https://support.google.com/webmasters/answer/83106?hl=en [support.google.com]

[jpeuron]

Hi aakk9999!

I've actually been relatively active on this forum before but I had to create a new account recently.

Thank you for your help. The website itself is relatively small, so I'll just make sure Google can find an empty robots.txt file there without restrictions.

I was convinced the "change of adress" tool had been removed from Search Console, thanks for pointing out that it is still accessible in the "settings"!

I have also told my clients to check the logs, and will keep doing so until all pages are gone.

Thanks again!