homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Is there such a thing as Allow: *

10+ Year Member

Msg#: 88 posted 12:13 am on Apr 3, 2001 (gmt 0)


Just a quick question regarding the robots.txt file. From everything I have read I gather that it is primarily used as excluding spiders from certain directories and specific pages. Is it possible to use it to allow spiders. I just want it on the site incase spiders look for it but I don't want to exclude the spiders from any areas in the site.
Therefore is there such a thing as the following?
User-agent: *
Allow: /

If so will this give access to all areas of the site.

Thanks in advance for anyone able to help :)



WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 88 posted 12:21 am on Apr 3, 2001 (gmt 0)

Yes, but not having one present is much safer. Some brain power challenged robots only look for the presence of a robots.txt and turn around if they find one. There is no "allow" in the official standard, but we've seen so many people use it, I think engines are allowing it.


Msg#: 88 posted 1:34 am on Apr 3, 2001 (gmt 0)

I agree. I don't have a robots.txt on my favorite site and spiders are always throwing parties there. Today, UA: ScoutMaster ( was the guest of honor. Maybe a new one???

BTW- The site is well indexed by most search engines, big and small. I don't even have to resubmit anymore. Makes my job easier.


10+ Year Member

Msg#: 88 posted 2:38 am on May 24, 2001 (gmt 0)

being the good little seo that i am, i receintly began actually putting something in the robots.txt that we had. just for clarification, are you saying that this could be bad? that, unless necessary, you should not have a robots.txt?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved