homepage Welcome to WebmasterWorld Guest from 107.21.187.131
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Dynamic URL's and robots.txt
cgydosh




msg:3568165
 1:56 am on Feb 7, 2008 (gmt 0)

If I have the following robots.txt file:

User-Agent: *
Disallow: /myfile.php

myfile.php is actually a dynamic url

will this block all of the following

myfile.php?id=1
myfile.php?id=2
myfile.php?id=3
...

Or do I need to do something else to block all of those?
Thanks for the help

 

ZydoSEO




msg:3568618
 3:42 pm on Feb 7, 2008 (gmt 0)

It should block

/myfile.php?id=1
/myfile.php?id=2
/myfile.php?id=3

as well as /myfile.php.

If you wanted it to block only /myfile.php but allow crawling URLs with /myfile.php followed by some query string parameter then you could use:

User-Agent: *
Disallow: /myfile.php$

Basically, if you don't specify a '$' on the end of the disallow, there is an implied wildcard ('*') on the end of the disallowed URL.

If you want to block all URLs regardless of where they live in your web directory structure if they contain the string 'myfile.php' then you could use:

User-Agent: *
Disallow: /*myfile.php

There is a lot of info on robots.txt if you simply search on it at G... wikipedia, G webmaster tools, etc. all have articles on its use.

[edited by: ZydoSEO at 3:45 pm (utc) on Feb. 7, 2008]

snowbirdcity




msg:3625971
 8:16 pm on Apr 13, 2008 (gmt 0)

Thanks ZydoSEO. Your post solved one of my problems. :)

"Basically, if you don't specify a '$' on the end of the disallow, there is an implied wildcard ('*') on the end of the disallowed URL."

I needed that $ to block a dynamic url.

burnseo




msg:3688563
 3:37 pm on Jul 2, 2008 (gmt 0)

Thanks ZydoSEO.

I have found your answer to be quite helpful as well =)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved