Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

How to setup robots.txt?

6:07 am on Jun 27, 2008 (gmt 0)

New User

10+ Year Member

joined:June 27, 2008
votes: 0


I know I would probably upload the text file to my ftp public, but what goes into the rebot.txt file? Correct me if I'm wrong on that first part please..

Thanks for any comments,


10:12 am on June 27, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 28, 2002
votes: 0

Hi Cameron, welcome to W.W.

as a first step, I would rename the "rebot.txt" to "robots.txt".

And a second step could be to type "robots.txt" into the search field of www.google.com, with plenty of information coming back then.

Good luck and kind regards,

10:23 am on June 27, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 27, 2002
votes: 0

There is some good info here [w3.org]

Do be shy, it's OK to look at other sites robots.txt files. www.example.com/robots.txt will open it up.

Welcome and Enjoy