Forum Moderators: goodroi
A client is launching a new site (website A) shortly that will have 30,000 data records (content pages etc) which will be searchable within the site. These records are being fed in from another website (website B) on the condition that website A does not compete against website B in the search engine results for any of the content in these records.
The client wants to know if blocking the content pages on webstite A from the search engines using a robots.txt file could cause them to be penalised given that it is an entirely new site and it might be deemed that they are tyring to do something untoward.
One other question is whether this content feed would be likely to been seen as mirror content by search engines, given that it will reflect content on website B. However, the structure of the 2 websites will be completely different and there will be plenty of additional content on website A.
Many thanks.
The client wants to know if blocking the content pages on webstite A from the search engines using a robots.txt file could cause them to be penalised
If I correctly understand, the client intends to use robots.txt to stop spiders from indexing the content pages on site A. If that's correct, then there wouldn't be any SE-levied penalty, but the pages wouldn't be indexed by the SEs, kind of a self-imposed shot in the foot.
whether this content feed would be likely to been seen as mirror content by search engines, given that it will reflect content on website B
Similarly, if site A's pages are not indexed they can't be seen as mirror content by the SEs.
Note that I might still be in need of another cup of coffe and my interpretation of your question can be way off.
