| 4:20 am on Jan 12, 2001 (gmt 0)|
What exactly is the robots.txt file used for?
| 12:36 am on Jan 13, 2001 (gmt 0)|
The robots.txt file (and the robots) META tag are used to
tell robots what they should and should not access.
Most robots follow the robots exclusion standard, which can be found at
The idea of limiting a robots crawling is that you might not want all pages indexed (any inexed page can be an entrypoint to your site). You wouldn't want to start your visit at the feedback page, or perhaps you dont want robots to index your pdf docs which you keep in a certain dir. Or there might be a section that's password protected, eg.
| 3:42 pm on Jan 13, 2001 (gmt 0)|
Robots is good for blocking off high profile directories. If it specific pages you want to block, I'd use a meta tag No Index tag - far more effective. Engines (like Google) will check for a robots once a month, whereas the meta tag is read every page read.
| 5:04 pm on Jan 13, 2001 (gmt 0)|
High profile meaning what ?
Forgive me..sometimes I'm retarded.
| 11:58 pm on Jan 13, 2001 (gmt 0)|
*shrug* That's why WE used it.
| 12:40 am on Jan 14, 2001 (gmt 0)|
I guess I better rename my IP delivery folder something other than "Super-Secret Search Engine Redirection Script" before I write that robots.txt file, huh?