Hello! In robots.txt files in Joomla...by default I see that images directory is NOT allowed to be indexed...but below it , there's a "stories" sub-directory which contains lots of graphic images.So would Google index "stories" subdirectory ? or because a parent directory (images) is marked as not allowed to be indexed, then the robots would NOT index any subdirectories underneath it ?
Disallow: /images/
that's how it appears in Joomla's default robots.txt file.
So I added also disallow: /images/stories
BTW, do robots mind the spaces? I mean in the original default Joomla robots.txt file there're no spaces...like:
User-agent: *Disallow:.......
I changed that to:
User-agent: * Disallow:.......
Also...after that...tor/
Disallow: /cache/Disallow: /components/Disallow:..........
You see, disallow has no space with a previous word...
Is it OK?
I get an error in Googlebot:
In Webmaster tools...:
Line 16: / Syntax not understood
Text of http://example.com/robots.txt
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
Disallow: /images/stories/
As you see...I disallowed a subdirectory after disallowing a parent directory...
and after I used robots.txt generator, I got this:
User-agent: *
Disallow: /components/
Disallow: /libraries/
Disallow: /images/
Disallow: /modules/
Disallow: /administrator/
Disallow: /xmlrpc/
Allow: /
Disallow: /cache/
Disallow: /language/
Disallow: /includes/
Disallow: /installation/
Disallow: /images/stories/
Disallow: /media/
Disallow: /plugins/
Disallow: /tmp/
Disallow: /templates/
That doesn't make any sense to me...why some directories are disallowed and some are allowed-disallowed ? Or...maybe it means that the default on the site is to allow everything, except what's disallowed...but then why that allow command stands in between blocks and not right in the beginning?
[edited by: goodroi at 12:54 pm (utc) on July 30, 2009]
[edit reason] Please no urls [/edit]
