homepage Welcome to WebmasterWorld Guest from 174.129.163.183
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Why use Robots - Again?
What are the benefits of using Robots.
oltc




msg:1528473
 3:55 pm on Mar 18, 2004 (gmt 0)

I have been doing SEO for sometime now, and I have had
good results. But I have been hearing about robots.txt
for years. I just don't understand the benefits of
them. So please help with some answers:

1. Does the use of Robots.txt help search engines
give you a higher ranking and why?

2. A lot of the research I have found says that
Robots disallow access to this file or directory.
But how does this help me?

3. Does anyone have the basic syntax of the robots
entries?

Thanks For your help.

 

ncw164x




msg:1528474
 4:16 pm on Mar 18, 2004 (gmt 0)

To create a robots.txt file
Create a text file using a Word Processor or HTML editor using the required coding as examples below
Save the file as robots.txt
Upload the robots.txt file to the root directory using your FTP software in ACSII mode

Examples
To exclude all robots from parts of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /misc/sitestats/

Exclude a specific spider from parts of the server
User-agent:slurp.so/
Disallow: /cgi-bin/
Disallow: /secure/
Disallow: /products/
Disallow:/misc/sitestats/

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot/2.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

Hope this helps

ncw164x

Mardi_Gras




msg:1528475
 4:35 pm on Mar 18, 2004 (gmt 0)

> 1. Does the use of Robots.txt help search engines
give you a higher ranking and why?

I guess it could, if it stopped a robot from finding stuff you didn't want found, but that is not really the purpose.

> 2. A lot of the research I have found says that
Robots disallow access to this file or directory.
But how does this help me?

That is exactly what robots.txt is used for - to exclude robots from certain files or folders. That can help you by keeping material you don't want indexed out of other people's hands.

Of course, there are robots that don't obey robots.txt, which calls for protection from .htacccess.

There is a lengthy discussion of .htaccess here:

[webmasterworld.com...]

added - your may want to view the robots.txt file for this site to get some ideas about how it is used - [webmasterworld.com...] /added

jdMorgan




msg:1528476
 4:48 pm on Mar 25, 2004 (gmt 0)

Google does support extensions [google.com] to the Standard for Robots Exclusion.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved