homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Gold Sponsor 2015!
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Wild chars *.* allowed in "Disallow" of Robots.txt?
or anything else like *.txt, etc.

10+ Year Member

Msg#: 386 posted 10:33 pm on May 19, 2004 (gmt 0)


I need to disallow all the files in the root directory, but allow the sub-directories. So, is it allowed (and does it work) to write the following in robots.txt file?

User-agent: *
Disallow: /*.*

Most of these files are txt. So, alternatively, will the following work also?

User-agent: *
Disallow: /*.txt

Thank you.



WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 386 posted 12:26 am on May 20, 2004 (gmt 0)


Welcome to WebmasterWorld [webmasterworld.com]!

Google supports extensions to the Standard for Robots Exclusion, but they are the only ones I know of. See their Wbemaster Help pages for more info.

If possible, re-arrange your directory structure into two branches - one branch for files you want indexed, and another for files you don't want indexed. Then you can exclude robots from entire directories, without relying on a technique that only Google supports.

The Standard was invented in simpler times, and is not very flexible. This should be considered when initially designing your sites.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved