Welcome to WebmasterWorld Guest from 54.147.212.98

Forum Moderators: goodroi

Message Too Old, No Replies

Help to rightly create robots.txt

   
4:37 am on Sep 10, 2010 (gmt 0)

5+ Year Member



Hi.

I have dynamic urls like this.

mydomain.com/?pg=login
mydomain.com/?pg=reguser
mydomain.com/?pg=aboutus
mydomain.com/?pg=termsofuse

When the page is requested for ex. mydomainname.com/?pg=login index.php include login.php file.

some of the urls are converted to static url like
mydomain.com/aboutus.html
mydomain.com/termsofuse.html

I need to allow index mydomainname.com/aboutus.html, mydomainname.com/termsofuse.html
and disallow mydomainname.com/?pg=login, mydomainname.com/?pg=reguser, please help to manage my robots.txt file.

I have also mydomainname.com/posted.php?details=50 (details can have any number) which I converted to mydomainname.com/details/50.html
I need also to allow all this type of urls.
8:51 am on Sep 10, 2010 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



robots.txt is not your technical solution here.
you should make sure all your internal links refer to the canonical urls.
(such as http://example.com/aboutus.html and http://example.com/details/50.html)
you should implement RewriteRules such that any requests for the dynamic urls are externally redirected to the static urls.
you should also implement RewriteRules so that any requests for the static urls are internally rewritten to the dynamic urls.
12:39 pm on Sep 11, 2010 (gmt 0)

5+ Year Member



But I don't need to convert all my dynamic urls to static.
I just want part of the to be converted.
12:22 am on Sep 12, 2010 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



in that case your RewriteRule/RewriteCond directives should include/exclude rewrites/redirects as required and you should also use the canonical static or dynamic urls for your internal navigation.
8:49 am on Sep 14, 2010 (gmt 0)

5+ Year Member



I don't need redirect any pages.

And please clearly explain what I need to do.