Forum Moderators: open
It spidered nearly 500 pages on the dev server and only 5 on my production server. This is NOT good. I have updated the robots.txt files to keep it off the development server (other spiders don't appear to be following its rules though)
I want to make sure the next crawl is done properly on the right server. What should I do? How can I get googlebot to stop spidering my devlopment server and spider my production server instead?