Page is a not externally linkable
hohmaster - 8:37 pm on Jan 24, 2001 (gmt 0)
User-agent: * I have read about it somewhere, but I forgot it :-( Could anyone help me?
I have a problem with my robots.txt. In my site there is a directory with lot of very similar files there. I want to exclude these files from SE robots. The problem is that there are subdirectories in this folder as well, and I don't want them to be excluded. To change the structure of the site is also complicated.
What should I use?
Disallow:/mydirectory/ OR
Disallow:/mydirectory