Welcome to WebmasterWorld Guest from 54.159.19.75

Forum Moderators: goodroi

Message Too Old, No Replies

Why use Robots - Again?

What are the benefits of using Robots.

     
3:55 pm on Mar 18, 2004 (gmt 0)

New User

10+ Year Member

joined:Mar 18, 2004
posts:2
votes: 0


I have been doing SEO for sometime now, and I have had
good results. But I have been hearing about robots.txt
for years. I just don't understand the benefits of
them. So please help with some answers:

1. Does the use of Robots.txt help search engines
give you a higher ranking and why?

2. A lot of the research I have found says that
Robots disallow access to this file or directory.
But how does this help me?

3. Does anyone have the basic syntax of the robots
entries?

Thanks For your help.

4:16 pm on Mar 18, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


To create a robots.txt file
Create a text file using a Word Processor or HTML editor using the required coding as examples below
Save the file as robots.txt
Upload the robots.txt file to the root directory using your FTP software in ACSII mode

Examples
To exclude all robots from parts of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /misc/sitestats/

Exclude a specific spider from parts of the server
User-agent:slurp.so/
Disallow: /cgi-bin/
Disallow: /secure/
Disallow: /products/
Disallow:/misc/sitestats/

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot/2.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

Hope this helps

ncw164x

4:35 pm on Mar 18, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 27, 2002
posts:1422
votes: 0


> 1. Does the use of Robots.txt help search engines
give you a higher ranking and why?

I guess it could, if it stopped a robot from finding stuff you didn't want found, but that is not really the purpose.

> 2. A lot of the research I have found says that
Robots disallow access to this file or directory.
But how does this help me?

That is exactly what robots.txt is used for - to exclude robots from certain files or folders. That can help you by keeping material you don't want indexed out of other people's hands.

Of course, there are robots that don't obey robots.txt, which calls for protection from .htacccess.

There is a lengthy discussion of .htaccess here:

[webmasterworld.com...]

added - your may want to view the robots.txt file for this site to get some ideas about how it is used - [webmasterworld.com...] /added

4:48 pm on Mar 25, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Google does support extensions [google.com] to the Standard for Robots Exclusion.

Jim