General Guide to Robots.txt

is there one here :)

[rj87uk]

Hi,

I am creating a website and have read about robot.txt etc and after reading alot i have become very confused!

can anyone point me in the direction of a good Guide?

Or is there one here?

I basicly want robots to go through all my site :)

I looked some robot.txt software but, again slightly confused!

Any help at all will be helpfull!

p.s. is it now necessary for you to have robots.txt?

[DaveAtIFG]

Try this [searchengineworld.com].

If you want to permit any robot to spider your entire site, you don't need a robots.txt. Robots.txt is exclusionary, if no robots.txt file is presented to spiders, they assume unfettered access. However, each spider that requests the missing robots.txt file will generate a 404 in your logs.

To create an appropriate robots.txt, use a plain text editor (NOT a word processor!) and create a file that contains these two lines:
User-agent: *
Disallow:

The wild card in the first line indicates that subsequent lines apply to ANY spider. The second line "disallows nothing," so your entire site is spidered.

Upload the file using your FTP program's ASCII transfer mode, to your site's root directory.

Not much to it really... :)

[ogletree]

It is not necessary to have one but it would save you some 404 errors if you would just put a blank text file there. You only need a robots.txt file if you have stuff you don't want spidered.

[rj87uk]

Thanks :D

I dont have a log as such, i use a free 1
(stat counter)

Its not a big site just wanting to know when it gets crawled and how often etc....

rj