How robot.txt works

Robot.txt

[walmslei]

Hi

I am recently tought to optimize my site. I visited www.google/webmasters/tools/sitemap

I added my site to sitemap. but its having some errors. Anyway Google have a tool for chacking robot.txt. for my site www.mysite/robot.txt,its giving 404(not found) messsage. I like to know do i need to create a robot.txt file by my own or it creates automatically?

[treeline]

If you want to BLOCK googlebot or other spiders from looking at your site, robots.txt is how you do it.

If you'd like google (and others) to index your entire site you don't need a robots.txt file.

robots.txt will not be created automatically. If you want it, you'll have to make it. It's fine that it's 404 Not Found, as long as you WANT all the pages to be found. If you need to hide some pages, then create one.

[jdMorgan]

And note that the file must be named robots.txt, not robot.txt

Jim