Forum Moderators: Robert Charlton & goodroi
I think you should spend some more time to find out what else you might have done that is fishy or that you actually have a penalty. Is your website completely deindexed or has your traffic gone to zero? How much traffic were you getting before? How much now? Remember that traffic fluctuations are very common. A site receiving just 100 visitors daily might go to just 10 visitors daily without any penalty at all (just because the SERPS change for one or two keywords). Yes it is a 90% drop in traffic but is more likely to happen. On the other hand, if a site with 10,000 visitors daily loses 90% of its traffic then it is probably a penalty.
So don't change anything for the time being and spend some time in finding out what really happened.
So my robots.txt file looks like this:
<%
$h = explode('.',$_SERVER['HTTP_HOST']);
$x = count($h);
$robot_host = $h[$x-2].'.'.$h[$x-1];
if (file_exists($_SERVER['DOCUMENT_ROOT'].'/robots.'.$robot_host.'.txt')) {
header("Content-Type: text/plain");
echo '# robots.'.$robot_host.'.txt'."\n";
echo '# host '.$_SERVER["SERVER_NAME"]."\n";
readfile($_SERVER['DOCUMENT_ROOT'].'/robots.'.$robot_host.'.txt');
} else {
header("HTTP/1.1 404 Not Found");
echo '<html><title>File Not Found</title><body><h4>File Not Found</h4></body></html>';
exit;
}
%>
and I have files like this:
robots.domain1.com.txt (contains my actual robots.txt instructions for the domain I want indexed)
robots.domain2.com.txt (contains a disallow * for every user agent)
And so on.
If someone knows a better way, I'm sure up for hearing it; I have quite a few clients with multiple domains that we're always trying to get sorted out.
The sites haven't been de-indexed, but I can determine by searching for specific phrases and page titles that they are imposing 150, 250, and 950 penalties depending on the specific pages. It has been this way for nearly a year.
I fully intend on 301 redirecting three of the domains to our primary domain, but this will take some time and I'm wondering if using "noindex, nofollow" on every page of the three lesser domains (effectively removing them from Google's index and eliminating cross-linking) will be enough to get Google to lift our penalty.
