homepage Welcome to WebmasterWorld Guest from 54.237.78.165
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt for directories with same name
vero

5+ Year Member



 
Msg#: 3965642 posted 8:40 pm on Aug 4, 2009 (gmt 0)

If I have in my robots file:

User-Agent: *
Disallow: /mypics/

Does this block every directory with the name "mypics" or only the first one off the root directory? For example, would everything in www.example.com/firstdirectory/seconddirectory/mypics/ be blocked too? Or do I have to specifically add

Disallow: /firstdirectory/seconddirectory/mypics/

Thanks

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3965642 posted 2:21 pm on Aug 5, 2009 (gmt 0)

The search engines will only block the example.com/mypics/ and will crawl example.com/anythinghere/mypics/.

If you want to block all directories that contain the letters "mypics" you need to use wildcards aka pattern matching. This is not officially part of the robots.xt protocol but it is supported by the big three search engines. Be careful some of the smaller crawlers do not support wildcards.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3965642 posted 8:25 pm on Aug 5, 2009 (gmt 0)

URLs are matched "from the left", so in this case, there is no match.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved