Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
Now I want to make sure the spiders follow the /catalog/ URLs and not the /cgi-bin/ URLs
If I disallow the /cgi-bin/, they will still follow the URLs that are mod_rewritten to /catalog/ even though the pre-rewritten URL is /cgi-bin/.
I am correct in this?
That's the reason I want to double check the robots.txt. I want to force the spiders into the new area, as the old URLs are tripping spiders up a bit due to the fact that they are dynamic. Unfortunatly, there is no way to change the old URLs right now, so I am stuck with this course of action.