robots.txt and mod_rewrite

just double checking

[hannamyluv]

Okay, I did a mod_rewrite that changes about half my URLs from
[example.com...]

to

[example.com...]

Now I want to make sure the spiders follow the /catalog/ URLs and not the /cgi-bin/ URLs

If I disallow the /cgi-bin/, they will still follow the URLs that are mod_rewritten to /catalog/ even though the pre-rewritten URL is /cgi-bin/.

I am correct in this?

TIA

[Birdman]

Have you changed the links in the HTML to point to the new, spider friendly, URLs? If not, you will need to or the bots will keep requesting them.

[hannamyluv]

The old URLs that siders are currently crawling will stay the same, due to the fact that I can not change them. I am creating new pages with the links to the modified URLs.

That's the reason I want to double check the robots.txt. I want to force the spiders into the new area, as the old URLs are tripping spiders up a bit due to the fact that they are dynamic. Unfortunatly, there is no way to change the old URLs right now, so I am stuck with this course of action.

[closed]

hannamyluv:

I am correct in this?

No. If they can't access

/cgi-bin/

, they can't follow the redirect.

[hannamyluv]

It isn't a redirect, though. It's a rewrite. Do they act the same?

[closed]

Redirect, rewrite, same difference in this case. :-)

Put it this way: the rewrite is in a room. If you don't allow access to the room, no one will actually know what's in it.