homepage Welcome to WebmasterWorld Guest from 54.166.120.175
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
robots.txt and joomla
member22

5+ Year Member



 
Msg#: 4517527 posted 3:03 pm on Nov 9, 2012 (gmt 0)

I use joomla for my website and automatically all those files are blocked is that good or bad, so I remove anything and if so why ?

I also added to my robots.txt files my email address ( is that useful, I am afraid google passes PR to the email address )
and a javascript: void (0) because I have tabs on my webpage ( is that useful )
as well as a .pdf ( is it also useful )

any comments ? does anything need to be changed or is it ok ?

Thank you,

 

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4517527 posted 5:45 pm on Nov 9, 2012 (gmt 0)

automatically all those files are blocked

All which files?

sunnyujjawal



 
Msg#: 4517527 posted 5:04 am on Nov 10, 2012 (gmt 0)

You are Preferred Member here, defiantly have more more knowledge than me. In case of robots.txt i understand only one thing, use it only if you want to block anything else delete it..Try it

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4517527 posted 7:22 am on Nov 10, 2012 (gmt 0)

I also added to my robots.txt files my email address ( is that useful, I am afraid google passes PR to the email address )
and a javascript: void (0) because I have tabs on my webpage ( is that useful )
as well as a .pdf ( is it also useful )

Say what now?

robots.txt is for naming directories on your site that you don't want to have crawled by good robots. Bad robots don't read robots.txt -- or they do read it and head straight for the listed directories -- so you have to 403 them.

I'm guessing there are certain directories used internally by joomla that it doesn't want to have crawled, so it comes with its own robots.txt to go with its own htaccess. This presumably overwrites your own pre-existing robots.txt, so I hope you kept a backup.

Rasputin

5+ Year Member



 
Msg#: 4517527 posted 9:07 am on Nov 10, 2012 (gmt 0)

Joomla basic robots.txt automatically blocks the images directory (or certainly did until very recently) so if you want images indexed you will need to remove that line from the file

member22

5+ Year Member



 
Msg#: 4517527 posted 2:36 pm on Nov 10, 2012 (gmt 0)

Here are the files

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4517527 posted 12:04 pm on Nov 11, 2012 (gmt 0)

the Disallow: directive matches URLs, left to right, that are to be excluded from crawling by "well-behaved bots".
if you have any content on urls that match those paths that you want indexed then you should change your robots.txt file accordingly.
however excluding a URL from being crawled does not prevent the URL from being indexed, it prevents the content from being indexed.
if you don't want either to appear in the index you must allow crawling of the URL and provide a noindex signal such as the meta robots noindex element for HTML documents or the X-Robots-Tag HTTP Response header for other content types.

Oimachi2

10+ Year Member



 
Msg#: 4517527 posted 11:14 am on Nov 12, 2012 (gmt 0)

I specialyze in Joomla.

YOu can just leave that file as is, I have dozens of sites that rank very well with the default robot.txt

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved