Why use Robots - Again?

What are the benefits of using Robots.

[oltc]

I have been doing SEO for sometime now, and I have had
good results. But I have been hearing about robots.txt
for years. I just don't understand the benefits of
them. So please help with some answers:

1. Does the use of Robots.txt help search engines
give you a higher ranking and why?

2. A lot of the research I have found says that
Robots disallow access to this file or directory.
But how does this help me?

3. Does anyone have the basic syntax of the robots
entries?

Thanks For your help.

[ncw164x]

To create a robots.txt file
Create a text file using a Word Processor or HTML editor using the required coding as examples below
Save the file as robots.txt
Upload the robots.txt file to the root directory using your FTP software in ACSII mode

Examples
To exclude all robots from parts of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /misc/sitestats/

Exclude a specific spider from parts of the server
User-agent:slurp.so/
Disallow: /cgi-bin/
Disallow: /secure/
Disallow: /products/
Disallow:/misc/sitestats/

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot/2.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

Hope this helps

ncw164x

[Mardi_Gras]

> 1. Does the use of Robots.txt help search engines
give you a higher ranking and why?

I guess it could, if it stopped a robot from finding stuff you didn't want found, but that is not really the purpose.

> 2. A lot of the research I have found says that
Robots disallow access to this file or directory.
But how does this help me?

That is exactly what robots.txt is used for - to exclude robots from certain files or folders. That can help you by keeping material you don't want indexed out of other people's hands.

Of course, there are robots that don't obey robots.txt, which calls for protection from .htacccess.

There is a lengthy discussion of .htaccess here:

[webmasterworld.com...]

added - your may want to view the robots.txt file for this site to get some ideas about how it is used - [webmasterworld.com...] /added

[jdMorgan]

Google does support extensions [google.com] to the Standard for Robots Exclusion.

Jim