Welcome to WebmasterWorld Guest from 54.211.17.91

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt for directories with same name

   
8:40 pm on Aug 4, 2009 (gmt 0)

5+ Year Member



If I have in my robots file:

User-Agent: *
Disallow: /mypics/

Does this block every directory with the name "mypics" or only the first one off the root directory? For example, would everything in www.example.com/firstdirectory/seconddirectory/mypics/ be blocked too? Or do I have to specifically add

Disallow: /firstdirectory/seconddirectory/mypics/

Thanks

2:21 pm on Aug 5, 2009 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The search engines will only block the example.com/mypics/ and will crawl example.com/anythinghere/mypics/.

If you want to block all directories that contain the letters "mypics" you need to use wildcards aka pattern matching. This is not officially part of the robots.xt protocol but it is supported by the big three search engines. Be careful some of the smaller crawlers do not support wildcards.

8:25 pm on Aug 5, 2009 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



URLs are matched "from the left", so in this case, there is no match.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month