Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
will my robots.txt that look like this:
prevent Google to spider all *.js and *.css files on my server? Also these ones that are in subdirectories? Is there any difference to:
Will this prevent Google to spider only all *.js and *.css files in my root directory?
"Disallow: *.js" will prevent the file named "*.js" being spidered.
"Disallow: /*.js" will prevent the directory named "*.js" being spidered.
<added>I nearly forgot! Welcome to WebmasterWorld, payday! :)</added>
Googlebot also understands some extensions to the robots.txt standard. Disallow patterns may include * to match any sequence of characters, and patterns may end in $ to indicate the end of a name. For example, to prevent Googlebot from crawling files that end in .gif, you may use the following robots.txt entry:
In a previous post someone stated that Googlebot is the only robot to accept these extensions, so using them will not keep other bots out of these pages.