Welcome to WebmasterWorld Guest from 54.163.168.15

Forum Moderators: goodroi

Robots.txt

Stopping robots visiting single dynamic page.

   
6:29 pm on Apr 21, 2003 (gmt 0)

10+ Year Member



Hi,

Hi, my site works on a MVC design i.e all requests go to /controller?requestType=theRequest

e.g. to get home page href="/controller?requestType=home"
to get links page href="/controller?requestType=links"

I now want to block robots going to some pages (eg a home link), so in my robots.txt:

User-agent: *
Disallow: /controller?requestType=home

Does the robots.txt standard and robots understand the parameter or will all requests to /controller be blocked?

Also, if i have links to external websites do they need to be in the robots.txt, or do the robots not traverse them by default?

Thank you

2:02 am on Apr 22, 2003 (gmt 0)

10+ Year Member



Hi donovana,

I don't believe the robots protocol can handle dynamic pages. However, I solved this problem on my website by using the mod_rewrite engine.

RewriteRule /foo/random.html$ /foo/?section=random [L]

In other words, any request for /foo/random.html will get redirected to the dynamic page without the browser (or spider) knowing the difference. Then I changed all my links pointing to the dynamic page to point to the static page and put the static page in my robots.txt.

If you want help on mod_rewrite, do a search here at WW for lots of information.

As for external links, your robots.txt applies only to your own domain.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month