Ok,
I have multiple sites, when I spider some of them using http they will spider but not using www, and others are just the opposite....... the robots.txt files r the same and all the information in the headers is the same.... Can someone explain why this would happen and how I can fix it.......
