Welcome to WebmasterWorld Guest from 54.163.52.98

Forum Moderators: goodroi

Message Too Old, No Replies

Where is the "Top Level"?

is this important these days?

     

chewy

7:47 pm on Mar 13, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



In the handy HTML Author's Guide (http://www.robotstxt.org/wc/exclusion-user.html) it says that robots.txt must be in the top level of a server's document space.

It also says that "if you rent space..." you usually aren't allowed to modify things at this level.

I and many others are using VHost or other approaches to host -- does this still apply?

C.

Romeo

9:03 pm on Mar 13, 2004 (gmt 0)

10+ Year Member



Generally, the robots.txt has to be in the document-root directory of the domain, which is defined in the web server's httpd.conf configuration file, and which is "/" from the internet web side's view, regardless if virtual or not. It is not tied to the root of the physical server's file system, so there is no problem with vhosts at all: just put the robots.txt file where the "/" of your domain is: if you have a "www.example.com/" site, simply put the robots.txt where it can be seen as "www.example.com/robots.txt".
On the opposite: if you have a site like "www.provider.com/private/my-dir/index.htm" where your own web site starts below some sub-directory, then you are lost, because your provider won't let you set a "www.provider.com/robots.txt" (and a "www.provider.com/private/my-dir/robots.txt" would be useless).

Regards,
R.

chewy

9:21 pm on Mar 13, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Good - that is what I thought.

Any comment on when this changed?

I suppose you can't trust much of what you read these days, 'specially on the 'net.

Thanks!

Romeo

9:46 pm on Mar 13, 2004 (gmt 0)

10+ Year Member



Chewy,
... it had always been this way, and the original author described the same thing -- just with other and fewer words, leaving room for some uncertaincies.

Regards,
R.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month