Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Removing "query-stringed" pages from Google

And other search engines using robotx.txt

8:55 pm on May 24, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 9, 2003
votes: 0

I have a few directories on my site related to my content management system that I don't want indexed by Google or the other SE's. Not that there's any harm in having them indexed; it's just that they're always redirected to my generic login page, which is thus indexed several times. I'd like to use robots.txt to exclude those directories since they serve no useful purpose to the SE's.

However, the directories are generated dynamically from time to time as modifications to the CMS are made. The only thing they'll have in common is the first eight letters of the directory name. After those first eight letters will be maybe eight or ten random numbers. (Bad for SEO, I know, but it's a third-party script...)

Is there any way to simply exclude those directories, categorically, using robots.txt? Remember I don't know what they'll be in the future, so I need to exclude them based only on the first eight characters of the directory name. Can that be done, and if so, how?

Also, given that these pages are already indexed, is there a safe way to remove them without affecting the rankings of my other pages?



2:09 pm on May 25, 2004 (gmt 0)

New User

10+ Year Member

joined:May 21, 2004
votes: 0

Your best bet would be to create another URL element before the dynamic portion, like this:

Before: somesite.com/directory/jhg8668765

After: somesite.com/directory/dynamic/jhg8668765

Then you can ban (in robots.txt) access to somesite.com/directory/dynamic/


2:23 pm on May 25, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
votes: 0

Try putting on your redirect page the meta <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
(or on your login page). This will stop them either indexing, or following, the dynamic directory.

(Assuming your directory page uses an HTML redirect and not 301/302?)