Forum Moderators: open
We have several sites before and now decide to only concentrate on one site.
The reason is if have several sites then it is vesy easy to have too many duplicate contents.
How to do this at yahoo? Did yahoo have a removal tool to remove the indexed page at yahoo?
We already put robot.txt to ban all bots.
Robots.txt after the even is uncertain, especially if there are links to the sites you are winding down.
Always best to do the permanent thing, don't risk short cuts: then you are future proof.
Then 301 from www.domain1 to www.domain2 - and also build a useful and user friendly 404 page to catch those who got lost along the way.
If any links to old URLs can be transerred to new, that'd be good.
Be sure to test the 301 by typing in 'old' URLs.
There's no better way to deal with duplicate sites; the 301 is the only way to hold on to visiotrs, as well as preserve any benefits from the old sites.
Note - if any of the old sites had Google penalties (ie had been delisted), DO NOT forward the damage with a 301.
is the following correct robots.txt to ban googlebot and yahoo sluep, can put two robots at one line? As this duplciated site still have some MSN traffic so we decide to ban google and yahoo.
*****************************************
User-agent: googlebot, Slurp
Disallow: /
*****************************************
Thanks,
If you do it wrong, it may work now, but that does not mean it always will.
An extra line or two really won't hurt.
User-agent: googlebot
Disallow: /
User-agent: Slurp
Disallow: /
or this better?
User-agent: googlebot
User-agent: Slurp
Disallow: /
User-agent: Slurp
Disallow: /
If you want to ban all, then:
User-agent: *
Disallow: /
