Forum Moderators: Robert Charlton & goodroi

Best way to stop Googlebot from crawling old AMP subdomain?

         

guarriman3

11:12 pm on Sep 2, 2025 (gmt 0)

10+ Year Member Top Contributors Of The Month



I run a large website with around 500,000 URLs under "example.com". Years ago, I also had an AMP version of the site, which created another 500,000 URLs under "amp.example.com" (one AMP for each canonical URL).

At some point I realized that having 1M URLs in total was hurting my crawl budget, so I decided to return 410 Gone for every URL under "amp.example.com" in order to get Google to forget them.

I’ve checked with tools like Ahrefs and confirmed that there are no external backlinks pointing to any of the AMP URLs.

The issue is that Googlebot is still crawling thousands of URLs from "amp.example.com" every day.

Now I’m debating between three options:

1) Keep returning 410 Gone for every AMP URL.
2) Remove the DNS A record for "amp.example.com", so the host no longer exists (NXDOMAIN).
3) A hybrid approach: keep returning 410 for 6–9 months, and then remove the DNS A record afterwards.

My main goal is to optimize crawl budget so that Googlebot focuses on crawling and ranking the main domain (example.com).

What do you think is the best option among these three?

tangor

12:38 am on Sep 3, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Wait.

That's it. g will never forget a URL it has met and will CONTINUE to try for YEARS (decades, actually), no matter what you do. g eventually gets the 410 message. So... wait.

Won't change your crawl budget... that's not really in your control in the first place. As said above, g will eventually get the message.

markRg

11:10 pm on Sep 14, 2025 (gmt 0)

Top Contributors Of The Month



Do you get traffic on the AMP version of your site? Maybe you could set up a 301 redirect to example.com

Brett_Tabke

6:21 pm on Sep 15, 2025 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Like MarkRg said, why no 301 it to decent pages on the main site?

tangor

12:57 am on Sep 17, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If the intent is to back off from 1m urls to 500k, why are those pages still active? Delete.

If the amp domain has no further use... see above. In which case a 301 is not necessary.

404 works GREAT!

All this is a concern for "crawl budget". You have no control over that, other than how many pages you offer. g will crawl at their time and for x-amount of pages, might crawl the SAME pages several times before moving on to whatever else is available and there's NOT A THING you can do about that---other than REDUCING what CAN BE CRAWLED. MAKE IT DISAPPEAR and then IGNORE all the hits that 404 (if DELETED AND DOMAIN GONE YOU WILL NEVER SEE ANYWAY) when g CONTINUES to call for URLS they have met and won't forget. NOT YOUR PROBLEM.

</forgive-my-rant>