Forum Moderators: phranque
User-agent: PITA_bot
Disallow: /whatever/bla*/
User-agent: PITA_bot
Disallow: /whatever/bla/*
User-agent: PITA_bot
Disallow: /whatever/bla*
The top one says to PITA_bot "Don't index any directory which is in the whatever file which starts with 'Bla'...
It seems to me the second one says "Don't index dirs underneath /whatever/bla/" which would still mean the page which sits at:
[yada.com...]
would be spidered.
The third one ... ?
Disallow: The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html.