risto - 12:37 pm on Dec 5, 2003 (gmt 0)

Hello Strike,

and wellcome. Search engine spiders are trying to find this file first. It tells them what they can and cannot spider. If it does not exist they think that everything on the website is spider food.

With robots.txt file the webmaster can ask spiders not to go to given directories or given files. Most robots follow the instructions but not all.