Forum Moderators: goodroi

Message Too Old, No Replies

mod_rewrite and robots.txt

         

jewels

10:19 am on May 1, 2005 (gmt 0)



My site was created with php. On my homepage index.php I have many links and I rewrote them using mod_rewrite so that:

[example.com...]

is written statically as:

[example.com...]

-----------------------------------
Options +FollowSymLinks
RewriteEngine on
RewriteRule form/(.*)/(.*)/$ /form.php?$1=$2
---------------------------------------------

Now on the inner page: [example.com...]

I have some links such as:
[example.com...]
These links I dont want crawled and are not rewritten into static pages.

My robots.txt file is defined as below:

User-agent: *
Disallow: /login.php
Disallow: /form.php
Disallow: /logout.php
Disallow: /signup.php
Disallow: /pwdremind.php
Disallow: /contact.php
Disallow: /priv_policy.php
Disallow: /disclaimer.php
Disallow: /terms_use.php

I was wondering if by disallowing form.php I will not be crawled on my statically created pages like:

[example.com...]

Will the above page be crawled by all robots?

[edited by: ThomasB at 2:58 pm (utc) on May 3, 2005]
[edit reason] example [/edit]

ThomasB

3:03 pm on May 3, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



jewels, first of all welcome to WebmasterWorld!

The robots.txt you posted should work out fine and the rewritten urls should be indexed by the search engines.