Forum Moderators: goodroi
The way of thinking is that peopel can look into your rorbots,txt fiel to see what directories or files are excluded. The do now then where to look for.
Is this right or not?
Is this right or not?
No its not right because if security is an issue then you should not publish content without requiring proper user authentication (via password or otherwise).
robots.txt exclusion was meant to be used to prevent robots indexing useless data, rather than excluding robots from content that is useful.
