Welcome to WebmasterWorld Guest from 54.147.20.131

Forum Moderators: open

Message Too Old, No Replies

robot.txt

     

almonday

6:53 pm on Feb 14, 2000 (gmt 0)



I want to create a robot.txt file that will do the following.

Disallow certain pages to ALL Spiders and then disallow some other pages to only one of the spiders. i.e.

moreinfo.htm not allowed for any spiders

moreinfo-alt.htm not allowed for Excite.

Also in the User-agent: what excatly am I putting there? Would it be scooter or Scooter/2.0 G.R.A.B. V1.1.0?

Thanks

DaveAtIFG

8:25 pm on Feb 14, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



A full robots.txt spec is at webcrawler. Half of it is Greek to me but there are examples in section 4 that are pretty simple.

Also a syntax checker is at robots syntax
and this is very handy.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month