Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Wild chars *.* allowed in "Disallow" of Robots.txt?

or anything else like *.txt, etc.



10:33 pm on May 19, 2004 (gmt 0)

10+ Year Member


I need to disallow all the files in the root directory, but allow the sub-directories. So, is it allowed (and does it work) to write the following in robots.txt file?

User-agent: *
Disallow: /*.*

Most of these files are txt. So, alternatively, will the following work also?

User-agent: *
Disallow: /*.txt

Thank you.


12:26 am on May 20, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


Welcome to WebmasterWorld [webmasterworld.com]!

Google supports extensions to the Standard for Robots Exclusion, but they are the only ones I know of. See their Wbemaster Help pages for more info.

If possible, re-arrange your directory structure into two branches - one branch for files you want indexed, and another for files you don't want indexed. Then you can exclude robots from entire directories, without relying on a technique that only Google supports.

The Standard was invented in simpler times, and is not very flexible. This should be considered when initially designing your sites.



Featured Threads

Hot Threads This Week

Hot Threads This Month