Welcome to WebmasterWorld Guest from

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google and mod_rewrite

using mod_rewrite with safety

3:56 am on Jun 23, 2005 (gmt 0)

New User

10+ Year Member

joined:July 3, 2004
votes: 0

I've made some SEO changes in my website and now, I'm using mod_rewrite. Do I've to remove all dynamic indexed pages from Google?

if yes, is this the right code for robots.txt file :

User-agent: Googlebot
Disallow: /*?

thanx in advance.

9:06 pm on July 2, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
votes: 0

Yes, having the old query-string urls expire from Google is a good idea -- otherwise, the rewrite is only introducing duplicate urls for the same content - right?

The syntax you have for Googlebot is what Google always recommended in the past, although I can't find it on their newly organized webmaster information at present. Everyone who used it reported that it worked, but you should know that the use of a wildcard in a robots.txt rule is a specific syntax just for Googlebot. Wildcards are not part of the robots.txt standard.