Welcome to WebmasterWorld Guest from 3.227.249.234

Forum Moderators: goodroi

Mistake in robots.txt during a site merge, caused a big drop

     
7:30 pm on Sep 24, 2019 (gmt 0)

New User

joined:Sept 24, 2019
posts: 2
votes: 0


Hi guys

We recently merged an ecommerce site into another, and all the configs and redirect instructions on the webserver level were set up correctly. 302 first to test, and then 301.

Since we used a tool to trigger a message to the users redirected and landing on the new page, we had to include UTM-parameters. This also to analyze the redirected traffic coming in.

Example
https://www.oldpage.com --> (301) https://www.newpage.com/?utm_source=old-page&utm_medium=referral


Now what we forgot, is that our robots.txt on both sites had the following instruction
Disallow: /*?*


And suddenly we were blocked in Search Console. We now have added Allow-instructions, so everything seems to be OK.

Million-dollar question is how much do you think we will suffer from this mistake? We merged September 2nd and discovered the error Sept 23rd. So that's 3 weeks of "off the grid" for a great e-commerce domain ranking high on a huge amount of keywords in our marked.

Thanks a lot in advance.

Best,
Christian
8:18 pm on Sept 24, 2019 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4509
votes: 348


Hi Christian and Welcome to WebmasterWorld [webmasterworld.com]

Things like this happen more often than you might think. Details. I would expect full recovery shortly and suggest it may help if you use the URL Inspection tool in your GSC account to let Google see that the problem was transient and has been repaired. You don't need to "inspect" every URL to get it moving.
8:50 pm on Sept 24, 2019 (gmt 0)

New User

joined:Sept 24, 2019
posts: 2
votes: 0


Thanks for a quick reply, not2easy:)
11:10 pm on Sept 24, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10457
votes: 1091


@Christian Evensen ... Welcome to Webmasterworld!

With robots.txt as a voluntary thing it is unlikely an error there will have long term effects. Wait three weeks and see what happens.