Forum Moderators: martinibuster
Problem is, Google now lists the sub-category directory pages under both the original site and virtual domain URLs. My AdSense revenues have tanked as computer-oriented ads increasingly show up where sub-category-specific ads should be (and used to be).
Queries for the sub-directory coming to the original URL have been redirected to the new virtual domain via the .htaccess file. I've tried to track down all the relative links from the "old" professional-site domain and make them absolute links to the new virtual domain; I'm requesting outside sites to link only to the new virtual domain, but I still seem to be losing ground.
So here's the question: can I put a robots.txt file in root telling Google to search everything but the sub-category directory; and another robots.txt file in the sub-category directory telling Google to go ahead and search everything there? Will this do the trick?
The logic is that Google will "stop at the door" of the sub-category directory when approaching via the old URL (root), but will be welcomed into the sub-cat directory when approaching via the new virtual domain.
Will this result in Google's seeing that there is only one set of pages there, and that I'm not trying to cheat?
Thanks for your help!
So here's the question: can I put a robots.txt file in root telling Google to search everything but the sub-category directory; and another robots.txt file in the sub-category directory telling Google to go ahead and search everything there? Will this do the trick?
Are we talking about the mediapartners robot? From what I was able to deduce from someone reporting that certain member only pages in a site {that might have been off-limits via a robots.txt file} the mediapartners bot might not be well behaved. Being that the bot goes only to the page that called it {by someone loading a page} and does not "crawl" in the way a normal spider does, maybe it does not follow the robots.txt file all the time or at all.
If the pages in the subdomain have been crawled already and are going to be refreshed by just going back to them every few days, then I think that it will be difficult to break the cycle since the bot will still consider the sub-category as part of the whole.
This is just speculation.
I have a support tag open with AdSense on this but their responses have been generally unhelpful: mildly positive but without details or results, generally clueless, and sometimes absurd.
But responses to my pleas for help from Google were not particularly encouraging.
AdSense tech support said "We were able to fix it." Well, maybe they did and maybe I did. I don't think I'll ever know whose actions fixed it.
Google tech support sent me this, which is a fairly good example of Microsoftesque double-speak: "We would not recommend using the robots.txt system you have described in your email. If you would prefer that [the sub-directory] not appear in the Google index. We would suggest creating a robots.txt file for that page. "
It's not a page it's a whole directory/folder; and what they're describing would be the opposite of a solution to my problem: it would result in no indexing at all rather than mistaken indexing, which was my problem.
It looks like an answer generated by a young, eager, inexperienced person with too many tech support requests, too little time, and no patience, incentive or capacity to actually understand what is wrong and recommend a viable solution.
But why am I surprised?
