Editing Robots.txt to block parameters

[serenoo]

I own an affiliation website where I cannot change a lot because the script is from the vendor.
But I can edit the robots.txt file.
My problem is that I have two pages with the same content:
www.mywebsiteexample.com/
www.mywebsiteexample.com/?lc=en

is there a way to block the second page by robots.txt?
Disallow: /?lc will work?

[goodroi]

it depends on how your website is setup. i have a feeling that the /?lc=en is a mirror of your index page. if that is the case you might want to try using a canonical tag. also make sure no links point to the duplicate page.

depending on your situation you could also use robots.txt wildcards aka pattern matching.

[g1smd]

Disallow: /*?lc=en

will also do it.

The * is the crucial character, as it will also work for /index.php?lc=en too.

[serenoo]

Just do it g1smd. Thanks, it works and it has already been cancelled by webmaster google tool. But I only added
Disallow: /?lc=en

without *

[phranque]

what happens when you request the default directory index document with that query string?
for example if your index document is index.php, do you 301 redirect www.example.com/index.php?lc=en to www.example.com/?lc=en or does it resolve with a 200 OK?

[spina45]

- Disallow: /*?lc=en -

Can you also...

Disallow: /*?

To disallow multiple in one shot?

[g1smd]

The rule is still potentially incomplete.

Disallow: /?lc=en

does not block requests for

/index.php?lc=en

for example.

That's another reason why

Disallow: /*?lc=en

was suggested.

[g1smd]

Disallow: /*?

will disallow all requests for URLs with parameters.

[spina45]

Webmaster Tools has a parameter handling page.

1) Can wildcards, or simply "?" be input?
2) Is it better to use their tool? Robots.txt? Both?

[g1smd]

Using

robots.txt

(note case) works for all search engines and is preferred.

[spina45]

Got it! Thanks! And thanks also for a post you made back in December re: .htaccess. I just came across it -- really helped me out. Thanks for all your valuable posts.

[spina45]

Is there any syntax to disallow all URLs that (mistakenly) have double slashes like this: www.domain.com/catagory//productname.html ?

I was trying add a slash at the end of: category and mistakenly added two just as Googlebot came by. The // has been corrected but I'd like Google to stop looking for it.

[g1smd]

Disallow: *//

would stop those URLs being crawled.
That won't help remove those URLs from the SERPs all that fast.

This code in your

.htaccess

file would be far more effective:

RewriteRule // - [G]

[spina45]

Thank you!

I had tried *// but the robots.txt testing area in GWT didn't disallow it. I appreciate the .htaccess code. Panda kicked my butt and I'm trying everything.