Welcome to WebmasterWorld Guest from 54.196.145.173

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt

Stopping robots visiting single dynamic page.

     
6:29 pm on Apr 21, 2003 (gmt 0)

New User

10+ Year Member

joined:Apr 21, 2003
posts:4
votes: 0


Hi,

Hi, my site works on a MVC design i.e all requests go to /controller?requestType=theRequest

e.g. to get home page href="/controller?requestType=home"
to get links page href="/controller?requestType=links"

I now want to block robots going to some pages (eg a home link), so in my robots.txt:

User-agent: *
Disallow: /controller?requestType=home

Does the robots.txt standard and robots understand the parameter or will all requests to /controller be blocked?

Also, if i have links to external websites do they need to be in the robots.txt, or do the robots not traverse them by default?

Thank you

2:02 am on Apr 22, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 18, 2002
posts:131
votes: 0


Hi donovana,

I don't believe the robots protocol can handle dynamic pages. However, I solved this problem on my website by using the mod_rewrite engine.

RewriteRule /foo/random.html$ /foo/?section=random [L]

In other words, any request for /foo/random.html will get redirected to the dynamic page without the browser (or spider) knowing the difference. Then I changed all my links pointing to the dynamic page to point to the static page and put the static page in my robots.txt.

If you want help on mod_rewrite, do a search here at WW for lots of information.

As for external links, your robots.txt applies only to your own domain.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members