homepage Welcome to WebmasterWorld Guest from 54.145.172.149
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Question concerning Disallow and *
augustwind

5+ Year Member



 
Msg#: 4004407 posted 5:58 pm on Oct 9, 2009 (gmt 0)

We have multiple folders in multiple locations where a subfolder is "popups", none of which we want crawled.
In the robots.txt file, would the following be valid, then?
disallow: */popups

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4004407 posted 12:25 pm on Oct 13, 2009 (gmt 0)

The big 3 search engines will accept it and will block all subfolders that are named popups. You are using wildcard aka pattern matching which technically it is not an official part of the robots.txt protocol. Some of the smaller engines which may have trouble with this unofficial work around.

Jonesy

5+ Year Member



 
Msg#: 4004407 posted 11:09 pm on Oct 18, 2009 (gmt 0)

I thought is was good enough to use:
disallow: popups/
to cover _all_ "popups" sub-directories on a site.

bigcat1967

5+ Year Member



 
Msg#: 4004407 posted 1:44 am on Oct 20, 2009 (gmt 0)

I think good is right - I believe you have to use a wildcard to cover everything (*)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved