Welcome to WebmasterWorld Guest from 50.19.0.90

Forum Moderators: goodroi

Message Too Old, No Replies

Help to rightly create robots.txt

     
4:37 am on Sep 10, 2010 (gmt 0)

New User

5+ Year Member

joined:Feb 21, 2008
posts: 9
votes: 0


Hi.

I have dynamic urls like this.

mydomain.com/?pg=login
mydomain.com/?pg=reguser
mydomain.com/?pg=aboutus
mydomain.com/?pg=termsofuse

When the page is requested for ex. mydomainname.com/?pg=login index.php include login.php file.

some of the urls are converted to static url like
mydomain.com/aboutus.html
mydomain.com/termsofuse.html

I need to allow index mydomainname.com/aboutus.html, mydomainname.com/termsofuse.html
and disallow mydomainname.com/?pg=login, mydomainname.com/?pg=reguser, please help to manage my robots.txt file.

I have also mydomainname.com/posted.php?details=50 (details can have any number) which I converted to mydomainname.com/details/50.html
I need also to allow all this type of urls.
8:51 am on Sept 10, 2010 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10542
votes: 8


robots.txt is not your technical solution here.
you should make sure all your internal links refer to the canonical urls.
(such as http://example.com/aboutus.html and http://example.com/details/50.html)
you should implement RewriteRules such that any requests for the dynamic urls are externally redirected to the static urls.
you should also implement RewriteRules so that any requests for the static urls are internally rewritten to the dynamic urls.
12:39 pm on Sept 11, 2010 (gmt 0)

New User

5+ Year Member

joined:Feb 21, 2008
posts: 9
votes: 0


But I don't need to convert all my dynamic urls to static.
I just want part of the to be converted.
12:22 am on Sept 12, 2010 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10542
votes: 8


in that case your RewriteRule/RewriteCond directives should include/exclude rewrites/redirects as required and you should also use the canonical static or dynamic urls for your internal navigation.
8:49 am on Sept 14, 2010 (gmt 0)

New User

5+ Year Member

joined:Feb 21, 2008
posts: 9
votes: 0


I don't need redirect any pages.

And please clearly explain what I need to do.