Allow directory/File in Robots.txt

[jebernier]

According to the latest Google Webmaster suggestion, I'm trying to allow the crawling of .css & .js files in my website.

My .css is in a /modules directory and I don't want Google to crawl everything in that directory. How do I specify just a certain file(s) in that directory and have Google ignore the rest?

[lucy24]

Google, unlike many search engines, recognizes the
Allow:
directive. (For all I know, they invented it.) So you can Disallow a directory and then Allow one file within it.

If the difference is in extensions, you can also Disallow (or Allow) site-wide by extension, like
Disallow: *.midi

[jebernier]

Thanks

[not2easy]

One suggestion - before you upload a new robots.txt, test it in GWT to be sure it does what you want it to do. I have tried testing it after upload and they are still on the cached version so you might as well use their test function to get it working the way you want before you upload it.

[jebernier]

Thanks not2easy