Welcome to WebmasterWorld Guest from 54.225.58.238

Forum Moderators: goodroi

Message Too Old, No Replies

Dynamic URL's and robots.txt

     
1:56 am on Feb 7, 2008 (gmt 0)

New User

5+ Year Member

joined:Jan 14, 2008
posts: 4
votes: 0


If I have the following robots.txt file:

User-Agent: *
Disallow: /myfile.php

myfile.php is actually a dynamic url

will this block all of the following

myfile.php?id=1
myfile.php?id=2
myfile.php?id=3
...

Or do I need to do something else to block all of those?
Thanks for the help

3:42 pm on Feb 7, 2008 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 5+ Year Member

joined:Nov 11, 2007
posts:769
votes: 1


It should block

/myfile.php?id=1
/myfile.php?id=2
/myfile.php?id=3

as well as /myfile.php.

If you wanted it to block only /myfile.php but allow crawling URLs with /myfile.php followed by some query string parameter then you could use:

User-Agent: *
Disallow: /myfile.php$

Basically, if you don't specify a '$' on the end of the disallow, there is an implied wildcard ('*') on the end of the disallowed URL.

If you want to block all URLs regardless of where they live in your web directory structure if they contain the string 'myfile.php' then you could use:

User-Agent: *
Disallow: /*myfile.php

There is a lot of info on robots.txt if you simply search on it at G... wikipedia, G webmaster tools, etc. all have articles on its use.

[edited by: ZydoSEO at 3:45 pm (utc) on Feb. 7, 2008]

8:16 pm on Apr 13, 2008 (gmt 0)

New User

5+ Year Member

joined:Sept 29, 2007
posts:7
votes: 0


Thanks ZydoSEO. Your post solved one of my problems. :)

"Basically, if you don't specify a '$' on the end of the disallow, there is an implied wildcard ('*') on the end of the disallowed URL."

I needed that $ to block a dynamic url.

3:37 pm on July 2, 2008 (gmt 0)

New User

5+ Year Member

joined:Mar 21, 2008
posts:13
votes: 0


Thanks ZydoSEO.

I have found your answer to be quite helpful as well =)