Welcome to WebmasterWorld Guest from 54.161.64.174

Forum Moderators: goodroi

Message Too Old, No Replies

Why use Robots - Again?

What are the benefits of using Robots.

     
3:55 pm on Mar 18, 2004 (gmt 0)

10+ Year Member



I have been doing SEO for sometime now, and I have had
good results. But I have been hearing about robots.txt
for years. I just don't understand the benefits of
them. So please help with some answers:

1. Does the use of Robots.txt help search engines
give you a higher ranking and why?

2. A lot of the research I have found says that
Robots disallow access to this file or directory.
But how does this help me?

3. Does anyone have the basic syntax of the robots
entries?

Thanks For your help.

4:16 pm on Mar 18, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



To create a robots.txt file
Create a text file using a Word Processor or HTML editor using the required coding as examples below
Save the file as robots.txt
Upload the robots.txt file to the root directory using your FTP software in ACSII mode

Examples
To exclude all robots from parts of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /misc/sitestats/

Exclude a specific spider from parts of the server
User-agent:slurp.so/
Disallow: /cgi-bin/
Disallow: /secure/
Disallow: /products/
Disallow:/misc/sitestats/

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot/2.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

Hope this helps

ncw164x

4:35 pm on Mar 18, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> 1. Does the use of Robots.txt help search engines
give you a higher ranking and why?

I guess it could, if it stopped a robot from finding stuff you didn't want found, but that is not really the purpose.

> 2. A lot of the research I have found says that
Robots disallow access to this file or directory.
But how does this help me?

That is exactly what robots.txt is used for - to exclude robots from certain files or folders. That can help you by keeping material you don't want indexed out of other people's hands.

Of course, there are robots that don't obey robots.txt, which calls for protection from .htacccess.

There is a lengthy discussion of .htaccess here:

[webmasterworld.com...]

added - your may want to view the robots.txt file for this site to get some ideas about how it is used - [webmasterworld.com...] /added

4:48 pm on Mar 25, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Google does support extensions [google.com] to the Standard for Robots Exclusion.

Jim

 

Featured Threads

Hot Threads This Week

Hot Threads This Month