Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

General Guide to Robots.txt

is there one here :)



1:52 pm on Jul 6, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member


I am creating a website and have read about robot.txt etc and after reading alot i have become very confused!

can anyone point me in the direction of a good Guide?

Or is there one here?

I basicly want robots to go through all my site :)

I looked some robot.txt software but, again slightly confused!

Any help at all will be helpfull!

p.s. is it now necessary for you to have robots.txt?


2:17 am on Jul 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Try this [searchengineworld.com].

If you want to permit any robot to spider your entire site, you don't need a robots.txt. Robots.txt is exclusionary, if no robots.txt file is presented to spiders, they assume unfettered access. However, each spider that requests the missing robots.txt file will generate a 404 in your logs.

To create an appropriate robots.txt, use a plain text editor (NOT a word processor!) and create a file that contains these two lines:
User-agent: *

The wild card in the first line indicates that subsequent lines apply to ANY spider. The second line "disallows nothing," so your entire site is spidered.

Upload the file using your FTP program's ASCII transfer mode, to your site's root directory.

Not much to it really... :)


2:20 am on Jul 7, 2004 (gmt 0)

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member

It is not necessary to have one but it would save you some 404 errors if you would just put a blank text file there. You only need a robots.txt file if you have stuff you don't want spidered.


7:51 pm on Jul 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Thanks :D

I dont have a log as such, i use a free 1
(stat counter)

Its not a big site just wanting to know when it gets crawled and how often etc....



Featured Threads

Hot Threads This Week

Hot Threads This Month