homepage Welcome to WebmasterWorld Guest from 54.166.159.110
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Where is the "Top Level"?
is this important these days?
chewy

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 325 posted 7:47 pm on Mar 13, 2004 (gmt 0)

In the handy HTML Author's Guide (http://www.robotstxt.org/wc/exclusion-user.html) it says that robots.txt must be in the top level of a server's document space.

It also says that "if you rent space..." you usually aren't allowed to modify things at this level.

I and many others are using VHost or other approaches to host -- does this still apply?

C.

 

Romeo

10+ Year Member



 
Msg#: 325 posted 9:03 pm on Mar 13, 2004 (gmt 0)

Generally, the robots.txt has to be in the document-root directory of the domain, which is defined in the web server's httpd.conf configuration file, and which is "/" from the internet web side's view, regardless if virtual or not. It is not tied to the root of the physical server's file system, so there is no problem with vhosts at all: just put the robots.txt file where the "/" of your domain is: if you have a "www.example.com/" site, simply put the robots.txt where it can be seen as "www.example.com/robots.txt".
On the opposite: if you have a site like "www.provider.com/private/my-dir/index.htm" where your own web site starts below some sub-directory, then you are lost, because your provider won't let you set a "www.provider.com/robots.txt" (and a "www.provider.com/private/my-dir/robots.txt" would be useless).

Regards,
R.

chewy

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 325 posted 9:21 pm on Mar 13, 2004 (gmt 0)

Good - that is what I thought.

Any comment on when this changed?

I suppose you can't trust much of what you read these days, 'specially on the 'net.

Thanks!

Romeo

10+ Year Member



 
Msg#: 325 posted 9:46 pm on Mar 13, 2004 (gmt 0)

Chewy,
... it had always been this way, and the original author described the same thing -- just with other and fewer words, leaving room for some uncertaincies.

Regards,
R.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved