homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Help to rightly create robots.txt

 4:37 am on Sep 10, 2010 (gmt 0)


I have dynamic urls like this.


When the page is requested for ex. mydomainname.com/?pg=login index.php include login.php file.

some of the urls are converted to static url like

I need to allow index mydomainname.com/aboutus.html, mydomainname.com/termsofuse.html
and disallow mydomainname.com/?pg=login, mydomainname.com/?pg=reguser, please help to manage my robots.txt file.

I have also mydomainname.com/posted.php?details=50 (details can have any number) which I converted to mydomainname.com/details/50.html
I need also to allow all this type of urls.



 8:51 am on Sep 10, 2010 (gmt 0)

robots.txt is not your technical solution here.
you should make sure all your internal links refer to the canonical urls.
(such as http://example.com/aboutus.html and http://example.com/details/50.html)
you should implement RewriteRules such that any requests for the dynamic urls are externally redirected to the static urls.
you should also implement RewriteRules so that any requests for the static urls are internally rewritten to the dynamic urls.


 12:39 pm on Sep 11, 2010 (gmt 0)

But I don't need to convert all my dynamic urls to static.
I just want part of the to be converted.


 12:22 am on Sep 12, 2010 (gmt 0)

in that case your RewriteRule/RewriteCond directives should include/exclude rewrites/redirects as required and you should also use the canonical static or dynamic urls for your internal navigation.


 8:49 am on Sep 14, 2010 (gmt 0)

I don't need redirect any pages.

And please clearly explain what I need to do.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved