Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- How can I block strange URL's?


lucy24 - 7:59 pm on Nov 14, 2012 (gmt 0)


My site is wrote on CMS, so, I think, it uses queries.

It almost certainly uses queries behind the scenes, but are they part of the visible URL? Google should only index URLs that are seen by humans. So the second-simplest form is

RewriteCond %{THE_REQUEST} ^[A-Z]{3-9}\ [^\s?]+\?
RewriteRule .* - [F]

meaning: the visitor asked for something containing a query string. (The \s is because I have temporarily forgotten whether the referer counts as part of the complete request. Almost everything from a search engine will have a ? in the referer.)

The third-simplest form is the one you have to use if you've got any kind of on-site analytics, because then you yourself will be asking for things with a query string.

In the fourth-simplest form, any request containing a query string is forcibly redirected to the queryless form of the same URL.

Are there many different queries, or are they the same ones over and over? Your first post showed "PAGEN_1" twice. You can go into Webmaster Tools and tell google to ignore this parameter. But also make sure that the googlebot is not able to get pages in this form if they don't really exist.


Thread source:: http://www.webmasterworld.com/robots_txt/4518731.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com