Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: phranque
does NOT disallow URL requests like
so you'll need another rule like
but not all robots understand that terminology. If you do use it, you need to place it in a specific Googlebot section, and then you have to also duplicate all of the rules from the User-agent: * section into the User-agent: Googlebot section of the file. This is because Google reads only the most specific section of the robots.txt file that applies to it.