Do i need robot.txt on my site

robot.txt-is it needed & how to add it

[Amitbud]

Hi

Need to know if adding robot.txt to my website would actually make it better off with the search engines,and how do i make it and add it.

Help appreciated a lot

Bud

[directrix]

Welcome to WebmasterWorld!

As far as I know, the presence of robots.txt neither helps nor hinders with the search engines. You might want to add a minimal robots.txt to prevent lots of 404 return codes in your server logs.

The file below is minimal. Line 1 means that all robots are welcome. Line 2 means that no files or directories are excluded from indexing. These are spider defaults, so the file is functionally equivalent to having no robots.txt file.

User-agent: *
Disallow:

The official robots.txt spec at [robotstxt.org...] states that, "The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.", but I've never tried that approach.

[vincevincevince]

I don't think that Disallow on its own shows that you are allowed...

[directrix]

It does. The Disallow field is a partial URL that is to be excluded. From the spec: "Any empty value, indicates that all URLs can be retrieved."

[penders]

I think you only need a "robots.txt" file if you want to prevent some/all robots from indexing part/all of your site. If you want all robots to index everything they can get their 'hands' on (including your images), then I can't see any reason to have a robots.txt file.

Also, (I don't know if this happens for sure, but...) if I was a malicious robot I'd have a look at the robots.txt file, see what's in the Disallow: entry and then go off and have a look! It draws the robots attention to what you don't want attention to be drawn to! So don't try to hide private/secret content using a robots.txt file.