homepage Welcome to WebmasterWorld Guest from 54.211.235.255
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Wild chars *.* allowed in "Disallow" of Robots.txt?
or anything else like *.txt, etc.
CodeCrunch




msg:1527948
 10:33 pm on May 19, 2004 (gmt 0)

Hi,

I need to disallow all the files in the root directory, but allow the sub-directories. So, is it allowed (and does it work) to write the following in robots.txt file?

User-agent: *
Disallow: /*.*

Most of these files are txt. So, alternatively, will the following work also?

User-agent: *
Disallow: /*.txt

Thank you.

 

jdMorgan




msg:1527949
 12:26 am on May 20, 2004 (gmt 0)

CodeCrunch,

Welcome to WebmasterWorld [webmasterworld.com]!

Google supports extensions to the Standard for Robots Exclusion, but they are the only ones I know of. See their Wbemaster Help pages for more info.

If possible, re-arrange your directory structure into two branches - one branch for files you want indexed, and another for files you don't want indexed. Then you can exclude robots from entire directories, without relying on a technique that only Google supports.

The Standard was invented in simpler times, and is not very flexible. This should be considered when initially designing your sites.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved