Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Should Wordpress theme folders be disallowed with robots.txt?

         

n00b1

4:07 pm on Jan 10, 2012 (gmt 0)

10+ Year Member



< moved from another location >

Is it wise to allow Google to crawl about it the WP theme folders etc. which contain your CSS files? If you block access it prevents one of the 'instant previews' from loading correctly. Would this count against you or should these folders by robotted?

[edited by: tedster at 2:13 am (utc) on Jan 12, 2012]

tedster

2:21 am on Jan 12, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



As far I know, the preview bot doesn't act like regular googlebot - it does not obey robots.txt, in other words.

n00b1

11:25 am on Jan 12, 2012 (gmt 0)

10+ Year Member



It is the 'Pre-render Desktop Search Instant Preview' that is prevented from loading correctly if the folders containing the CSS elements etc. are robotted. The 'On-Demand' previews load fine regardless but the 'Pre-render' only appears with all the bells and whilstes if the folders are crawlable.