Forum Moderators: goodroi

Message Too Old, No Replies

Robots - Disallow All Except Images, JS & CSS

         

almo136

10:27 pm on Oct 5, 2016 (gmt 0)

10+ Year Member



Would the below in robots.txt prevent Google from indexing any of the pages but would still allow the images, JS and CS to be crawled and indexed (if they were linked to from another url)?

User-agent: *
Disallow: /
Allow /*.css
Allow /*.js
Allow /*.jpg
Allow /*.jpeg
Allow /*.gif
Allow /*.png

topr8

11:02 pm on Oct 5, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



in principle your code makes sense. however i'd possibly rewrite it as:
(note that google understands the * symbol in this context other bots might not)

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot
Disallow: /
Allow /*.css
Allow /*.js

it is good practice to put all your css files in a single folder and likewise your javascript files, in which case you could avoid using the wildcard character - so all bots could understand it.

User-agent: Googlebot-Image
Allow: /
User-agent: *
Disallow: /
Allow /css/
Allow /js/