Welcome to WebmasterWorld Guest from 54.198.77.172

Forum Moderators: goodroi

Message Too Old, No Replies

question about an empty robots.txt

     
5:01 pm on Mar 4, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 1, 2002
posts:444
votes: 0


i created an empty robots.txt (although i don't know really the purpose of it, but people here talk so much about it) and validated on searchengineworld, but there was an error:

Syntax check robots.txt on www.mydomain.com
ERROR There should be atleast 1 disallow line in any Robots.txt.

We're sorry, this robots.txt does NOT validate.
Warnings Detected: 0
Errors Detected: 1

so is it really an error?

5:49 pm on Mar 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


Do you mean empty by nothing at all in the file?

An example of a robots.txt file is as below

User-agent: *
Disallow: /cgi-bin/

The User-agent: * means you are not blocking or banning any spiders from accessing your site

The disallow is for any directories which you do not want to be spidered

If you do not have a robots.txt file it will make no difference to your site other than when a spider/robot visits and requests this file because it is not there it will put a 404 error in the server error log file

6:53 pm on Mar 4, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 1, 2002
posts:444
votes: 0


thank you ncw164x,

>Do you mean empty by nothing at all in the file?

yes.

i made some research and googleguy said here once that it would be better to have an empty robots.txt - even if you don't want to disallow any robot. don't ask my why - he just said it here, so i'll do it.

but what is an empty robots.txt, does it mean nothing in the file or is it this one here:

User-agent: *
Disallow:

7:09 pm on Mar 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


huhufrufru,

Either way is OK.

The "empty file" method is most useful for those who have difficulty uploading a robots.txt file for some reason. Sometimes, it's easier to just create a blank file on the server, and name it robots.txt.

The robots.txt code you posted to allow all robots is better.

The purpose of the blank or "allow all" robots.txt is simply to prevent a large number of 404 errors in your logs caused by robots trying to request robots.txt and not finding it.

Jim

8:36 pm on Mar 4, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 1, 2002
posts:444
votes: 0


ahhh, i see, thank you Jim :-)