Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Spider.txt


hohmaster - 8:37 pm on Jan 24, 2001 (gmt 0)


I have a problem with my robots.txt. In my site there is a directory with lot of very similar files there. I want to exclude these files from SE robots. The problem is that there are subdirectories in this folder as well, and I don't want them to be excluded. To change the structure of the site is also complicated.
What should I use?

User-agent: *
Disallow:/mydirectory/ OR
Disallow:/mydirectory

I have read about it somewhere, but I forgot it :-(

Could anyone help me?


Thread source:: http://www.webmasterworld.com/robots_txt/155.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com