Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
I am creating a website and have read about robot.txt etc and after reading alot i have become very confused!
can anyone point me in the direction of a good Guide?
Or is there one here?
I basicly want robots to go through all my site :)
I looked some robot.txt software but, again slightly confused!
Any help at all will be helpfull!
p.s. is it now necessary for you to have robots.txt?
If you want to permit any robot to spider your entire site, you don't need a robots.txt. Robots.txt is exclusionary, if no robots.txt file is presented to spiders, they assume unfettered access. However, each spider that requests the missing robots.txt file will generate a 404 in your logs.
To create an appropriate robots.txt, use a plain text editor (NOT a word processor!) and create a file that contains these two lines:
The wild card in the first line indicates that subsequent lines apply to ANY spider. The second line "disallows nothing," so your entire site is spidered.
Upload the file using your FTP program's ASCII transfer mode, to your site's root directory.
Not much to it really... :)