homepage Welcome to WebmasterWorld Guest from 50.19.172.0
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Help to rightly create robots.txt
mcnika

5+ Year Member



 
Msg#: 4199823 posted 4:37 am on Sep 10, 2010 (gmt 0)

Hi.

I have dynamic urls like this.

mydomain.com/?pg=login
mydomain.com/?pg=reguser
mydomain.com/?pg=aboutus
mydomain.com/?pg=termsofuse

When the page is requested for ex. mydomainname.com/?pg=login index.php include login.php file.

some of the urls are converted to static url like
mydomain.com/aboutus.html
mydomain.com/termsofuse.html

I need to allow index mydomainname.com/aboutus.html, mydomainname.com/termsofuse.html
and disallow mydomainname.com/?pg=login, mydomainname.com/?pg=reguser, please help to manage my robots.txt file.

I have also mydomainname.com/posted.php?details=50 (details can have any number) which I converted to mydomainname.com/details/50.html
I need also to allow all this type of urls.

 

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4199823 posted 8:51 am on Sep 10, 2010 (gmt 0)

robots.txt is not your technical solution here.
you should make sure all your internal links refer to the canonical urls.
(such as http://example.com/aboutus.html and http://example.com/details/50.html)
you should implement RewriteRules such that any requests for the dynamic urls are externally redirected to the static urls.
you should also implement RewriteRules so that any requests for the static urls are internally rewritten to the dynamic urls.

mcnika

5+ Year Member



 
Msg#: 4199823 posted 12:39 pm on Sep 11, 2010 (gmt 0)

But I don't need to convert all my dynamic urls to static.
I just want part of the to be converted.

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4199823 posted 12:22 am on Sep 12, 2010 (gmt 0)

in that case your RewriteRule/RewriteCond directives should include/exclude rewrites/redirects as required and you should also use the canonical static or dynamic urls for your internal navigation.

mcnika

5+ Year Member



 
Msg#: 4199823 posted 8:49 am on Sep 14, 2010 (gmt 0)

I don't need redirect any pages.

And please clearly explain what I need to do.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved