Welcome to WebmasterWorld Guest from 54.167.175.107

Forum Moderators: goodroi

Robots.txt

Is there such a thing as Allow: *

   
12:13 am on Apr 3, 2001 (gmt 0)

10+ Year Member



Hi,

Just a quick question regarding the robots.txt file. From everything I have read I gather that it is primarily used as excluding spiders from certain directories and specific pages. Is it possible to use it to allow spiders. I just want it on the site incase spiders look for it but I don't want to exclude the spiders from any areas in the site.
Therefore is there such a thing as the following?
User-agent: *
Allow: /

If so will this give access to all areas of the site.

Thanks in advance for anyone able to help :)

12:21 am on Apr 3, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Yes, but not having one present is much safer. Some brain power challenged robots only look for the presence of a robots.txt and turn around if they find one. There is no "allow" in the official standard, but we've seen so many people use it, I think engines are allowing it.
1:34 am on Apr 3, 2001 (gmt 0)



I agree. I don't have a robots.txt on my favorite site and spiders are always throwing parties there. Today, UA: ScoutMaster (62.133.27.210) was the guest of honor. Maybe a new one???

BTW- The site is well indexed by most search engines, big and small. I don't even have to resubmit anymore. Makes my job easier.

2:38 am on May 24, 2001 (gmt 0)

10+ Year Member



being the good little seo that i am, i receintly began actually putting something in the robots.txt that we had. just for clarification, are you saying that this could be bad? that, unless necessary, you should not have a robots.txt?
 

Featured Threads

Hot Threads This Week

Hot Threads This Month