Forum Moderators: goodroi

Message Too Old, No Replies

parameter page to be banned from crawling

directory.asp?

         

idolw

1:27 pm on Dec 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have one directory divided into 2 categories. One shall be indexed, the other not. The URL for the main directory page (where the user chooses from two main categories) is:
[mydomain.com...]

The category I want to be indexed and cached is:
[mydomain.com...]

The category I do not want to be indexed and cached is:
[mydomain.com...]

so will:

User-Agent: *
Disallow: /mydirectory.asp?cat=TheOneNotToBeCached

work for me? if not, what will?

idolw

11:12 am on Dec 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



anyone?

ThomasB

3:30 pm on Dec 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



idolw, first off it would be important to know what exactly the URL is. Currently your post is kind of confusing in my oppinion:

Disallow: /mydirectory.asp?cat=TheOneNotToBeCached
vs
[mydomain.com...]

Generally speaking robots.txt does not support parameters and therefor it might not work, though Google for example accepts wildcards in robots.txt files

idolw

1:18 pm on Dec 15, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



ThomasB,

thanks for your reply.
The URLs I want to ban are:
[mydomain.com...]