Welcome to WebmasterWorld Guest from 54.159.85.193

Forum Moderators: goodroi

Message Too Old, No Replies

Robots - Disallow All Except Images, JS & CSS

     
10:27 pm on Oct 5, 2016 (gmt 0)

Junior Member

5+ Year Member

joined:Sept 11, 2008
posts: 192
votes: 0


Would the below in robots.txt prevent Google from indexing any of the pages but would still allow the images, JS and CS to be crawled and indexed (if they were linked to from another url)?

User-agent: *
Disallow: /
Allow /*.css
Allow /*.js
Allow /*.jpg
Allow /*.jpeg
Allow /*.gif
Allow /*.png
11:02 pm on Oct 5, 2016 (gmt 0)

Senior Member

WebmasterWorld Senior Member topr8 is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 19, 2002
posts:3377
votes: 41


in principle your code makes sense. however i'd possibly rewrite it as:
(note that google understands the * symbol in this context other bots might not)

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot
Disallow: /
Allow /*.css
Allow /*.js

it is good practice to put all your css files in a single folder and likewise your javascript files, in which case you could avoid using the wildcard character - so all bots could understand it.

User-agent: Googlebot-Image
Allow: /
User-agent: *
Disallow: /
Allow /css/
Allow /js/
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members