Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
In clear words:
==> DOES NOT get crawled
==> DOES get crawled?
From this behaviour I would take the leap and say that:
/this.php will be crawled and indexed if your robots.txt has a directive to disallow: /this.php?
At least for Google, not sure how the other engines will take it. I checked the robots documentation as well and couldn't find any specific examples..
Google and Ask Jeeves have a behaviour which is different from most other spiders: If either of these spiders finds a link to a page, they will list the page, regardless of whether robots.txt disallows crawling of that page. If the page is disallowed, they won't crawl (fetch) it, but they will list it by URL. Other search engine spiders interpret a Disallow as meaning "don't mention this page at all," but the "listing" behaviour of Google and AJ is not explicitly defined by A Standard for Robots Exclusion; all it describes is fetching behaviour.
Yidaki brings up another grey-area question: Since a query string is not technically part of a URL (it is instead an argument passed to an agent at a specific URL), then is a robot expected/required to recognize different query string values as part of the URL for the purposes of matching a Disallow directive? My guess is that it is not a good idea to depend on any standard behaviour of different robots with respect to query strings. This may be another good argumant in favor of using URL rewriting to make dynamic URLs look like static ones.
Just some comments...
subcategory.php?param=16&subcat=blablaa ==> allow
subcategory.php?param=16 ==> allow
or even just page.php ==> allow
so dont be worry using param as long you feel that your pages will crawling by googlebot
i have more 2000 pages and still increasing, using php,including parameter.