homepage Welcome to WebmasterWorld Guest from 107.22.127.92
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Accredited PayPal World Seller

Visit PubCon.com
Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
Forum Library : Charter : Moderator: open

Deprecated - Search Engine Submission Forum

    
robot.txt
almonday




msg:703147		 6:53 pm on Feb 14, 2000 (gmt 0)

I want to create a robot.txt file that will do the following.

Disallow certain pages to ALL Spiders and then disallow some other pages to only one of the spiders. i.e.

moreinfo.htm not allowed for any spiders

moreinfo-alt.htm not allowed for Excite.

Also in the User-agent: what excatly am I putting there? Would it be scooter or Scooter/2.0 G.R.A.B. V1.1.0?

Thanks

 

DaveAtIFG




msg:703148		 8:25 pm on Feb 14, 2000 (gmt 0)

A full robots.txt spec is at webcrawler. Half of it is Greek to me but there are examples in section 4 that are pretty simple.

Also a syntax checker is at robots syntax
and this is very handy.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved