Forum Moderators: goodroi
I recently added a robots.txt file to my site, and told the bots to stay away from duplicate pages with different phone numbers that were already indexed(unfortunately we did this for campaign tracking). I also used no index, no follow on the same excluded pages. Msn was ok with this, but it looks like Yahoo slammed us. Is there a way to fix this dumb newbie move?
Thanks.
Usually bots are quite fast at applying changes to your robots.txt file. Nevertheless it might happen that there's some delay between reading/spidering them and actually seeing the difference in the index.
How many days were between the change of the robots.txt file and now?
My main concern in all of this was the potential duplicate content penalty, since pages were exact except for phone number differences.
Should I just drop the txt file altogether, or is the protocol to have one now?
