Welcome to WebmasterWorld Guest from 54.227.214.52

Forum Moderators: goodroi

question about an empty robots.txt

   
5:01 pm on Mar 4, 2003 (gmt 0)

10+ Year Member



i created an empty robots.txt (although i don't know really the purpose of it, but people here talk so much about it) and validated on searchengineworld, but there was an error:

Syntax check robots.txt on www.mydomain.com
ERROR There should be atleast 1 disallow line in any Robots.txt.

We're sorry, this robots.txt does NOT validate.
Warnings Detected: 0
Errors Detected: 1

so is it really an error?

5:49 pm on Mar 4, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Do you mean empty by nothing at all in the file?

An example of a robots.txt file is as below

User-agent: *
Disallow: /cgi-bin/

The User-agent: * means you are not blocking or banning any spiders from accessing your site

The disallow is for any directories which you do not want to be spidered

If you do not have a robots.txt file it will make no difference to your site other than when a spider/robot visits and requests this file because it is not there it will put a 404 error in the server error log file

6:53 pm on Mar 4, 2003 (gmt 0)

10+ Year Member



thank you ncw164x,

>Do you mean empty by nothing at all in the file?

yes.

i made some research and googleguy said here once that it would be better to have an empty robots.txt - even if you don't want to disallow any robot. don't ask my why - he just said it here, so i'll do it.

but what is an empty robots.txt, does it mean nothing in the file or is it this one here:

User-agent: *
Disallow:

7:09 pm on Mar 4, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



huhufrufru,

Either way is OK.

The "empty file" method is most useful for those who have difficulty uploading a robots.txt file for some reason. Sometimes, it's easier to just create a blank file on the server, and name it robots.txt.

The robots.txt code you posted to allow all robots is better.

The purpose of the blank or "allow all" robots.txt is simply to prevent a large number of 404 errors in your logs caused by robots trying to request robots.txt and not finding it.

Jim

8:36 pm on Mar 4, 2003 (gmt 0)

10+ Year Member



ahhh, i see, thank you Jim :-)
 

Featured Threads

Hot Threads This Week

Hot Threads This Month