homepage Welcome to WebmasterWorld Guest from 54.204.94.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Robots how to
Need help
maatrix




msg:398013
 7:40 am on Mar 28, 2001 (gmt 0)

Hi,
I`m new here and I need help whit the robots.txt
creation.(I`m a pure beginer)
Please can you tell me if when I disallow some directiry or file to googlebot if then are stoped
all google spiders or have I to write the full names of all hunderts of google spiders.
Thangs Maatrix

 

Son_House




msg:398014
 3:59 am on Mar 29, 2001 (gmt 0)

Welcome to WebmasterWorld maatrix!

As far as I know, you only have to use Googlebot. Now if the pages you put in your robots.txt are already indexed by Google, it may take a few months for Google to take them out.

User-agent: Googlebot
Disallow: /cgi-bin
Disallow: /images
Disallow: /doh.htm

This should keep Googlebot away from those parts of your site. If you want to keep all bots away from the same parts of your site, you can do this instead:

User-agent: *
Disallow: /cgi-bin
Disallow: /images
Disallow: /doh.htm

Here are some helpful links:
[info.webcrawler.com...]
[info.webcrawler.com...]
[webmasterworld.com...]

maatrix




msg:398015
 7:30 am on Mar 29, 2001 (gmt 0)

Thangs

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved