Disallowing Google robots to index in Joomla

[abrodski]

Hello!

I know that if I DON'T want a certain page to be indexed by spiders, then in Joomla MetaData Information -Robots field, I type-nofollow
Is there such a thing as noindex, nofollow ?
Or just nofollow will suffice? Any difference at all?

And the 2nd part of my question is...
In robots.txt files in Joomla...by default I see that images directory is NOT allowed to be indexed...but below it , there's a "stories" sub-directory which contains lots of graphic images.So would Google index "stories" subdirectory ? or because a parent directory (images) is marked as not allowed to be indexed, then the robots would NOT index any subdirectories underneath it ?

[spadilla]

Yes there is such a thing as using noindex and nofollow together (assuming you are taking meta tags). You can also noindex and rel="nofollow" on the URLs depending on your needs. The latter is more difficult in Joomla however. The difference between noindex and nofollow is explained here: [webmasterworld.com...]

IIRC the robots.txt in Joomla is written with a wildcard after the images directory which blocks the sub-directories. You might find it helpful to sign up with google webmaster tools and test your robots.txt there with different URLs.

[edited by: ergophobe at 5:05 pm (utc) on July 28, 2009]
[edit reason] typo (text -> test) [/edit]

[abrodski]

Disallow: /images/
that's how it appears in Joomla's default robots.txt file.
So I added also disallow: /images/stories
BTW, do robots mind the spaces? I mean in the original default Joomla robots.txt file there're no spaces...like:
User-agent: *Disallow:.......
I changed that to:
User-agent: * Disallow:.......
Also...after that...tor/
Disallow: /cache/Disallow: /components/Disallow:..........
You see, disallow has no space with a previous word...
Is it OK?

[ergophobe]

I don't know about whitespace in robots.txt - as spadilla said, you can test it in your Webmaster tools interface. But all examples I've seen include spaces after the colon.

As for the lack of space between the * and the Disallow, I suspect that is a matter of converting from a Unix filetype to Windows, which has different encoding for a new line.

And then the /images/stories disallow is superfluous because /images already disallows everything below that level.

Hope that helps.

[abrodski]

superfluous...but it wouldn't hurt to have it, would it?...or should I bother deleting it now after I wrote it already?

[abrodski]

Also...Google made me add a line of code for their robots tool to be functional.A meta tag...Is that OK to add it in index.php file of my current Joomla's template?

[ergophobe]

Superfluous but in no way harmful, so don't bother. Just for future reference.

I don't know Joomla templates, so I don't know where you would put it. Are you talking about the site owner verification for webmaster tools? I always find it easier to just use the file method (FTP a given file to the server and your done).

All sites in WMT will use the same filename for verification, so once you create the file (1 min) it takes 10 sec to FTP it to a new site. No messing with meta tags, no worry about editing templates. I find that simpler.

[abrodski]

what file? at Google they say that the meta tag with their verification code should be added in home page's HTML.
Since I don't have .html, but rather index.php, I just change index.php's code.And index.php file is in my current templates folder in Joomla!

[abrodski]

I probably should remove that code alltogether...not like their tools were useful to me so far...

[ergophobe]

The WMT can be quite useful once they've collected some data - such as searches where your page shows, but does not get clicked. No analytics you can put on your site could ever tell you that.

As for the file, you have to check the other radio button. You have the "meta tags" button checked on the verification page. Check the html file button and you'll get instructions.

You might find this useful
[google.com...]

[spadilla]

I agree with ergophobe. Uploading the blank html file named whatever they tell you to name it, is far easier than the other (default) verification method in Joomla unless you want to install a plugin, but why bother? Naming a txt file and uploading via FTP takes 2 seconds. Webmaster tools is worth it in my opinion.