Welcome to WebmasterWorld Guest from 23.22.207.70

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt and mod_rewrite

just double checking

     
1:20 pm on Mar 9, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 18, 2002
posts:2283
votes: 3


Okay, I did a mod_rewrite that changes about half my URLs from
[example.com...]

to

[example.com...]

Now I want to make sure the spiders follow the /catalog/ URLs and not the /cgi-bin/ URLs

If I disallow the /cgi-bin/, they will still follow the URLs that are mod_rewritten to /catalog/ even though the pre-rewritten URL is /cgi-bin/.

I am correct in this?

TIA

1:44 pm on Mar 9, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 22, 2002
posts:2546
votes: 0


Have you changed the links in the HTML to point to the new, spider friendly, URLs? If not, you will need to or the bots will keep requesting them.
2:35 pm on Mar 9, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 18, 2002
posts:2283
votes: 3


The old URLs that siders are currently crawling will stay the same, due to the fact that I can not change them. I am creating new pages with the links to the modified URLs.

That's the reason I want to double check the robots.txt. I want to force the spiders into the new area, as the old URLs are tripping spiders up a bit due to the fact that they are dynamic. Unfortunatly, there is no way to change the old URLs right now, so I am stuck with this course of action.

5:26 am on Mar 12, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 20, 2003
posts:255
votes: 0


hannamyluv:

I am correct in this?

No. If they can't access

/cgi-bin/
, they can't follow the redirect.
1:57 pm on Mar 13, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 18, 2002
posts:2283
votes: 3


It isn't a redirect, though. It's a rewrite. Do they act the same?
2:24 am on Mar 14, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 20, 2003
posts:255
votes: 0


Redirect, rewrite, same difference in this case. :-)

Put it this way: the rewrite is in a room. If you don't allow access to the room, no one will actually know what's in it.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members