Welcome to WebmasterWorld Guest from 34.204.191.31

Forum Moderators: goodroi

Message Too Old, No Replies

directory structure question

     
9:45 am on Jan 19, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 1, 2003
posts:145
votes: 0


I want to block some duplicate pages and I am slightly confused with directory issues with robot.txt.

My domain structure is
domainname.com/area/brochures/name.htm

If i wish to block these pages, is this correct?

User-Agent: *
Disallow: /area/brochures/

or can I use

User-Agent: *
Disallow: /brochures/

I have about 50 areas so if the first example is required will this be a problem?

thanks 4 any help?

4:04 pm on Jan 21, 2007 (gmt 0)

New User

10+ Year Member

joined:Jan 20, 2007
posts:37
votes: 0


Can anyone reply on this, I have the same question.
3:33 pm on Jan 22, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 1, 2003
posts:145
votes: 0


mmm, guess it was a stupid question as nobody answered it lol
5:41 pm on Jan 22, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 10, 2006
posts:160
votes: 0


I don't think this is a stupid question! Unfortunately I think you're going to have to do it the hardway, e.g.

User-Agent: *
Disallow: /area1/brochures/
Disallow: /area2/brochures/
Disallow: /area3/brochures/
etc.

I don't think that the wildcard syntax is that sophisticated with robots.txt.

7:43 pm on Jan 22, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 1, 2003
posts:145
votes: 0


thank 4 that, i was looking for a short cut, but I guess better to be safe than sorry :-)