| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
I want to create a robot.txt file that will do the following.
Disallow certain pages to ALL Spiders and then disallow some other pages to only one of the spiders. i.e.
moreinfo.htm not allowed for any spiders
moreinfo-alt.htm not allowed for Excite.
Also in the User-agent: what excatly am I putting there? Would it be scooter or Scooter/2.0 G.R.A.B. V1.1.0?
A full robots.txt spec is at webcrawler. Half of it is Greek to me but there are examples in section 4 that are pretty simple.
Also a syntax checker is at robots syntax
and this is very handy.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved