Forum Moderators: goodroi

Message Too Old, No Replies

Help with disallow

Help with disallow

         

SlugKing

9:40 pm on Sep 14, 2005 (gmt 0)

10+ Year Member



Hello, I am working on a yahoo store and found the following in the robots.txt file. The are a few more that were easily identified but I cant seem to figure out what these are? Can someone help?
Thanks

Disallow: /RT
Disallow: /TG
Disallow: /QS

Disallow: /I
Disallow: /wgl
Disallow: /P
Disallow: /ymix

Lord Majestic

9:42 pm on Sep 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



These disallow statements will be used by good bots that comply with robots.txt standard to NOT crawl URLs that start with specified strings.

Ie Disallow: /RT will prevent URL http://www.example.com/RT.htm from being retrieved.

SlugKing

10:18 pm on Sep 14, 2005 (gmt 0)

10+ Year Member



Thanks I got that part. The rt.html does not exist in the store. As a url IE example.com/rt.htm forwards to the index page as do all of the ones I noted. I assume they must be be part of the Yahoo store set up. I just cant figure out what they are?
Slugking

Lord Majestic

10:21 pm on Sep 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you have a URL that does such a vital function as redirecting to the homepage then it might be worth making sure its not excluded from indexing. On the other hand there might be considerations why to exclude it, it all depends on whether people who link to you use this form of URL or not.