Forum Moderators: goodroi

Message Too Old, No Replies

What's Robot.txt?

What's the importance of Robot.txt?

         

Lowkei

6:17 pm on Jul 30, 2005 (gmt 0)

10+ Year Member



Hello everybody,
I'm just wondering,what's actually a robot.txt?What is the use of it?To what i understand,robot.txt is a file that contenting instructions to SE spiders where to crawl throughout the site.Am i correct?
What's the importance of it actually?Will it affect any of the judgement from SE?And,how can one set up a robot.txt file?
I have a site without any robot.txt.Do you think i should build one?
Thanz in advance for the input :)

Lowkei

ThomasB

7:12 pm on Jul 30, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Lowkei, there's no real need to have a robots.txt file if you don't have anything on your site, that is not password protected, that spider should not crawl. You can find detailed information in the Robots.txt Tutorial [searchengineworld.com]

Lowkei

9:02 am on Jul 31, 2005 (gmt 0)

10+ Year Member



So that means I will not need a Robot.txt file if I don't have any file that is private,right?I see I see..Ok Thanz a lot pal!
The tutorials link do help!Thanks a lot:)

Lowkei