hi
i am new to robots.txt. i have no problems adding files and folders that should not be spidered.
however, i have a asp.net site which obvioulsy uses aspx.vb control files.
do i need to add entries in the robots.txt to stop the spidering of the vb files?
thanks guys