Forum Moderators: goodroi
will my robots.txt that look like this:
User-agent: Googlebot
Disallow: *.js
Disallow: *.css
prevent Google to spider all *.js and *.css files on my server? Also these ones that are in subdirectories? Is there any difference to:
User-agent: Googlebot
Disallow: /*.js
Disallow: /*.css
Will this prevent Google to spider only all *.js and *.css files in my root directory?
"Disallow: *.js" will prevent the file named "*.js" being spidered.
"Disallow: /*.js" will prevent the directory named "*.js" being spidered.
<added>I nearly forgot! Welcome to WebmasterWorld, payday! :)</added>
Maybe else someone has more detailed info?
Googlebot also understands some extensions to the robots.txt standard. Disallow patterns may include * to match any sequence of characters, and patterns may end in $ to indicate the end of a name. For example, to prevent Googlebot from crawling files that end in .gif, you may use the following robots.txt entry:
[pre]
User-Agent: Googlebot
Disallow: /*.gif$[/pre]
In a previous post someone stated that Googlebot is the only robot to accept these extensions, so using them will not keep other bots out of these pages.