Welcome to WebmasterWorld Guest from 54.166.102.61

Forum Moderators: goodroi

Dynamic URL's and robots.txt

   
1:56 am on Feb 7, 2008 (gmt 0)

5+ Year Member



If I have the following robots.txt file:

User-Agent: *
Disallow: /myfile.php

myfile.php is actually a dynamic url

will this block all of the following

myfile.php?id=1
myfile.php?id=2
myfile.php?id=3
...

Or do I need to do something else to block all of those?
Thanks for the help

3:42 pm on Feb 7, 2008 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



It should block

/myfile.php?id=1
/myfile.php?id=2
/myfile.php?id=3

as well as /myfile.php.

If you wanted it to block only /myfile.php but allow crawling URLs with /myfile.php followed by some query string parameter then you could use:

User-Agent: *
Disallow: /myfile.php$

Basically, if you don't specify a '$' on the end of the disallow, there is an implied wildcard ('*') on the end of the disallowed URL.

If you want to block all URLs regardless of where they live in your web directory structure if they contain the string 'myfile.php' then you could use:

User-Agent: *
Disallow: /*myfile.php

There is a lot of info on robots.txt if you simply search on it at G... wikipedia, G webmaster tools, etc. all have articles on its use.

[edited by: ZydoSEO at 3:45 pm (utc) on Feb. 7, 2008]

8:16 pm on Apr 13, 2008 (gmt 0)

5+ Year Member



Thanks ZydoSEO. Your post solved one of my problems. :)

"Basically, if you don't specify a '$' on the end of the disallow, there is an implied wildcard ('*') on the end of the disallowed URL."

I needed that $ to block a dynamic url.

3:37 pm on Jul 2, 2008 (gmt 0)

5+ Year Member



Thanks ZydoSEO.

I have found your answer to be quite helpful as well =)

 

Featured Threads

Hot Threads This Week

Hot Threads This Month