Forum Moderators: coopster

Message Too Old, No Replies

A PHP Website, mod_rewrite, and crawling.

How do the bots handle it?

         

fidibidabah

2:02 pm on Jun 17, 2004 (gmt 0)

10+ Year Member



I'm just curious, if my site is php based, and I use mod_rewrite to change page.html?page=content into content.html, how will the bots spider it? I know mod_rewrite is suppose to happen before anything else, but will the bots just see the link as content.html (assuming i use a bidirectional rewrite) and follow that link and end up at content.html, indexing it that way, without it running into dynamic anything?

Also, if I use a one way mod_rewrite, and leave the internal links in php, will they still follow it? If they do, they'd end up at the friendly link (technically), so is that what they'd index?

Just trying to figure out the path of least resistance, thank you.

CompressedAir

3:52 pm on Jun 18, 2004 (gmt 0)

10+ Year Member



If you are using mod_rewrite you want to put the links in your site in the form that you want the spiders to see. So, you should make your links like 'content.html' and NOT 'somepage.php?page=content.' If all your links are just regular .html links then that is how the SE's will see them. They won't know that you are rewriting the urls.
If you were to use the longer?var= urls, they might index them. I've seen some pages been indexed in this form. But why would you? If you have already setup mod_rewrite there is no point to keep using those ugly URLs!

Hope that helps :)