homepage Welcome to WebmasterWorld Guest from 54.227.34.0
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
question about an empty robots.txt
HuhuFruFru

10+ Year Member



 
Msg#: 17 posted 5:01 pm on Mar 4, 2003 (gmt 0)

i created an empty robots.txt (although i don't know really the purpose of it, but people here talk so much about it) and validated on searchengineworld, but there was an error:

Syntax check robots.txt on www.mydomain.com
ERROR There should be atleast 1 disallow line in any Robots.txt.

We're sorry, this robots.txt does NOT validate.
Warnings Detected: 0
Errors Detected: 1

so is it really an error?

 

ncw164x

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 17 posted 5:49 pm on Mar 4, 2003 (gmt 0)

Do you mean empty by nothing at all in the file?

An example of a robots.txt file is as below

User-agent: *
Disallow: /cgi-bin/

The User-agent: * means you are not blocking or banning any spiders from accessing your site

The disallow is for any directories which you do not want to be spidered

If you do not have a robots.txt file it will make no difference to your site other than when a spider/robot visits and requests this file because it is not there it will put a 404 error in the server error log file

HuhuFruFru

10+ Year Member



 
Msg#: 17 posted 6:53 pm on Mar 4, 2003 (gmt 0)

thank you ncw164x,

>Do you mean empty by nothing at all in the file?

yes.

i made some research and googleguy said here once that it would be better to have an empty robots.txt - even if you don't want to disallow any robot. don't ask my why - he just said it here, so i'll do it.

but what is an empty robots.txt, does it mean nothing in the file or is it this one here:

User-agent: *
Disallow:

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 17 posted 7:09 pm on Mar 4, 2003 (gmt 0)

huhufrufru,

Either way is OK.

The "empty file" method is most useful for those who have difficulty uploading a robots.txt file for some reason. Sometimes, it's easier to just create a blank file on the server, and name it robots.txt.

The robots.txt code you posted to allow all robots is better.

The purpose of the blank or "allow all" robots.txt is simply to prevent a large number of 404 errors in your logs caused by robots trying to request robots.txt and not finding it.

Jim

HuhuFruFru

10+ Year Member



 
Msg#: 17 posted 8:36 pm on Mar 4, 2003 (gmt 0)

ahhh, i see, thank you Jim :-)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved