Forum Moderators: goodroi

Message Too Old, No Replies

Help with blocking URLS

         

jaffstar

12:49 pm on Feb 14, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have 12 files that are causing problems with google crawling.

They are only 2 variations that are found on 6 different urls.

?variable=white and ?variable=black

But are found on for example:

www.domain.com/keyword-keyword1.php?variable=black
www.domain.com/keyword-keyword1.php?variable=white

Above is instance 1.

Instance 2, would say have keyword-keyword2.php?variable=white

DO I block:

dissallow:?variable=white Or the full path ? like :
dissallow:keyword-keyword1.php?variable=black

Quadrille

1:25 pm on Feb 14, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Probably better to use noindex, if there's any chance that links may appear from other places.

As a general policy, you need to try and reduce all URL duplication, it will always bite yer bum.

jaffstar

1:26 pm on Feb 14, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



So , what would the code be? would it include instance 1 or 2?

ZydoSEO

6:29 am on Feb 18, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



[google.com...]

According to Google

Disallow: /*?variable=white

will disallow any URL that begins with your domain name, followed by any string, followed by a string of the form '?variable=white', followed by any string.

So

Disallow: /*?variable=white
Disallow: /*?variable=black

should disallow all URLs containing '?variable=white' or '?variable=black'.

The above should disallow:

www.example.com/abc/def.asp?variable=white
www.example.com/abc/ghi.asp?variable=white
www.example.com/abc/ghi.asp?variable=whitewashthefence
www.example.com/abc/jkl.asp?variable=white&id=123
www.example.com/abc/jkl.asp?variable=white&id=123&shape=sqare

It would NOT disallow:

www.example.com/abc/jkl.asp?id=123&variable=white&shape=sqare
www.example.com/abc/jkl.asp?id=123&shape=sqare&variable=white

However, Disallow: /*?*variable=white shouldould.

According to Google, basically there is an implied wildcard at the end of the Disallowed pattern unless you use a '$' to indicate end of line.

[google.com...]

So

Disallow: /*?variable=white$

would disallow

www.example.com/abc/def.asp?variable=white
www.example.com/abc/ghi.asp?variable=white

but would allow:

www.example.com/abc/ghi.asp?variable=whitewashthefence
www.example.com/abc/jkl.asp?variable=white&id=123
www.example.com/abc/jkl.asp?variable=white&id=123&shape=sqare

I'm not sure, however, if all of the engines support this syntax.

[edited by: ZydoSEO at 7:08 am (utc) on Feb. 18, 2008]

jaffstar

7:17 am on Feb 18, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks for a very informative and helpful post :) I will add now and see if this helps cure the problem.