homepage Welcome to WebmasterWorld Guest from 54.211.138.180
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt
Is there such a thing as Allow: *
merlin78




msg:1526665
 12:13 am on Apr 3, 2001 (gmt 0)

Hi,

Just a quick question regarding the robots.txt file. From everything I have read I gather that it is primarily used as excluding spiders from certain directories and specific pages. Is it possible to use it to allow spiders. I just want it on the site incase spiders look for it but I don't want to exclude the spiders from any areas in the site.
Therefore is there such a thing as the following?
User-agent: *
Allow: /

If so will this give access to all areas of the site.

Thanks in advance for anyone able to help :)

 

Brett_Tabke




msg:1526666
 12:21 am on Apr 3, 2001 (gmt 0)

Yes, but not having one present is much safer. Some brain power challenged robots only look for the presence of a robots.txt and turn around if they find one. There is no "allow" in the official standard, but we've seen so many people use it, I think engines are allowing it.

Froggyman




msg:1526667
 1:34 am on Apr 3, 2001 (gmt 0)

I agree. I don't have a robots.txt on my favorite site and spiders are always throwing parties there. Today, UA: ScoutMaster (62.133.27.210) was the guest of honor. Maybe a new one???

BTW- The site is well indexed by most search engines, big and small. I don't even have to resubmit anymore. Makes my job easier.

crash




msg:1526668
 2:38 am on May 24, 2001 (gmt 0)

being the good little seo that i am, i receintly began actually putting something in the robots.txt that we had. just for clarification, are you saying that this could be bad? that, unless necessary, you should not have a robots.txt?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved