Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Wild chars *.* allowed in "Disallow" of Robots.txt?

or anything else like *.txt, etc.

10:33 pm on May 19, 2004 (gmt 0)

New User

10+ Year Member

joined:May 19, 2004
votes: 0


I need to disallow all the files in the root directory, but allow the sub-directories. So, is it allowed (and does it work) to write the following in robots.txt file?

User-agent: *
Disallow: /*.*

Most of these files are txt. So, alternatively, will the following work also?

User-agent: *
Disallow: /*.txt

Thank you.

12:26 am on May 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0


Welcome to WebmasterWorld [webmasterworld.com]!

Google supports extensions to the Standard for Robots Exclusion, but they are the only ones I know of. See their Wbemaster Help pages for more info.

If possible, re-arrange your directory structure into two branches - one branch for files you want indexed, and another for files you don't want indexed. Then you can exclude robots from entire directories, without relying on a technique that only Google supports.

The Standard was invented in simpler times, and is not very flexible. This should be considered when initially designing your sites.