Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
When using home directories under Apache servers in my [exit109.com...] directory,
if I want to exclude /~ghealton/tmp/ to I use /~ghealton/tmp/, or like
so many people seem to use, /tmp/ in the Disallow statement in ~ghealton/robots.txt? If it is /~ghealton/tmp/ then you have a nother common error to add to the list of common problems.
At this time I am using BOTH to assure all spiders will avoid my files.
Robots Exclusion Protocol [robotstxt.org]
Pointless robots.txt URLs
http //www w3.org/admin/robots.txt
http //www w3.org/~timbl/robots.txt
ftp //ftp w3.com/robots.txt
So, you need to provide the "/robots.txt" in the top-level of your URL space. How to do this depends on your particular server software and configuration.
For most servers it means creating a file in your top-level server directory. On a UNIX machine this might be...