homepage Welcome to WebmasterWorld Guest from 54.205.193.39
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Help to rightly create robots.txt
mcnika




msg:4199825
 4:37 am on Sep 10, 2010 (gmt 0)

Hi.

I have dynamic urls like this.

mydomain.com/?pg=login
mydomain.com/?pg=reguser
mydomain.com/?pg=aboutus
mydomain.com/?pg=termsofuse

When the page is requested for ex. mydomainname.com/?pg=login index.php include login.php file.

some of the urls are converted to static url like
mydomain.com/aboutus.html
mydomain.com/termsofuse.html

I need to allow index mydomainname.com/aboutus.html, mydomainname.com/termsofuse.html
and disallow mydomainname.com/?pg=login, mydomainname.com/?pg=reguser, please help to manage my robots.txt file.

I have also mydomainname.com/posted.php?details=50 (details can have any number) which I converted to mydomainname.com/details/50.html
I need also to allow all this type of urls.

 

phranque




msg:4199897
 8:51 am on Sep 10, 2010 (gmt 0)

robots.txt is not your technical solution here.
you should make sure all your internal links refer to the canonical urls.
(such as http://example.com/aboutus.html and http://example.com/details/50.html)
you should implement RewriteRules such that any requests for the dynamic urls are externally redirected to the static urls.
you should also implement RewriteRules so that any requests for the static urls are internally rewritten to the dynamic urls.

mcnika




msg:4200479
 12:39 pm on Sep 11, 2010 (gmt 0)

But I don't need to convert all my dynamic urls to static.
I just want part of the to be converted.

phranque




msg:4200669
 12:22 am on Sep 12, 2010 (gmt 0)

in that case your RewriteRule/RewriteCond directives should include/exclude rewrites/redirects as required and you should also use the canonical static or dynamic urls for your internal navigation.

mcnika




msg:4201540
 8:49 am on Sep 14, 2010 (gmt 0)

I don't need redirect any pages.

And please clearly explain what I need to do.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved