homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Where is the "Top Level"?
is this important these days?

 7:47 pm on Mar 13, 2004 (gmt 0)

In the handy HTML Author's Guide (http://www.robotstxt.org/wc/exclusion-user.html) it says that robots.txt must be in the top level of a server's document space.

It also says that "if you rent space..." you usually aren't allowed to modify things at this level.

I and many others are using VHost or other approaches to host -- does this still apply?




 9:03 pm on Mar 13, 2004 (gmt 0)

Generally, the robots.txt has to be in the document-root directory of the domain, which is defined in the web server's httpd.conf configuration file, and which is "/" from the internet web side's view, regardless if virtual or not. It is not tied to the root of the physical server's file system, so there is no problem with vhosts at all: just put the robots.txt file where the "/" of your domain is: if you have a "www.example.com/" site, simply put the robots.txt where it can be seen as "www.example.com/robots.txt".
On the opposite: if you have a site like "www.provider.com/private/my-dir/index.htm" where your own web site starts below some sub-directory, then you are lost, because your provider won't let you set a "www.provider.com/robots.txt" (and a "www.provider.com/private/my-dir/robots.txt" would be useless).



 9:21 pm on Mar 13, 2004 (gmt 0)

Good - that is what I thought.

Any comment on when this changed?

I suppose you can't trust much of what you read these days, 'specially on the 'net.



 9:46 pm on Mar 13, 2004 (gmt 0)

... it had always been this way, and the original author described the same thing -- just with other and fewer words, leaving room for some uncertaincies.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved