Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies


Is there such a thing as Allow: *

12:13 am on Apr 3, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:July 26, 2000
votes: 0


Just a quick question regarding the robots.txt file. From everything I have read I gather that it is primarily used as excluding spiders from certain directories and specific pages. Is it possible to use it to allow spiders. I just want it on the site incase spiders look for it but I don't want to exclude the spiders from any areas in the site.
Therefore is there such a thing as the following?
User-agent: *
Allow: /

If so will this give access to all areas of the site.

Thanks in advance for anyone able to help :)

12:21 am on Apr 3, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 111

Yes, but not having one present is much safer. Some brain power challenged robots only look for the presence of a robots.txt and turn around if they find one. There is no "allow" in the official standard, but we've seen so many people use it, I think engines are allowing it.
1:34 am on Apr 3, 2001 (gmt 0)

Full Member

joined:Jan 24, 2001
votes: 0

I agree. I don't have a robots.txt on my favorite site and spiders are always throwing parties there. Today, UA: ScoutMaster ( was the guest of honor. Maybe a new one???

BTW- The site is well indexed by most search engines, big and small. I don't even have to resubmit anymore. Makes my job easier.

2:38 am on May 24, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 26, 2001
votes: 0

being the good little seo that i am, i receintly began actually putting something in the robots.txt that we had. just for clarification, are you saying that this could be bad? that, unless necessary, you should not have a robots.txt?