Yidaki - 6:20 pm on Sep 20, 2003 (gmt 0) In clear words: Disallow: /this.php? -> /this.php?param=example
I want to allow googlebot to crawl my index page named index.php but i want to disallow index.php?param=example. So if i make a robots.txt entry Disallow: /index.php? will index.php still get crawled? I saw in google's own robots.txt [google.com] that they themself exlude /mac? but i assume this would allow /mac ...!? Is this valid? Can i use it? Couldn't find ANY info about it neither on google nor within the w3c specs ...
==> DOES NOT get crawled
==> DOES get crawled?
In clear words: