Welcome to WebmasterWorld Guest from 107.20.75.63

Forum Moderators: goodroi

Message Too Old, No Replies

Where is the "Top Level"?

is this important these days?

     
7:47 pm on Mar 13, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 7, 2003
posts:1048
votes: 0


In the handy HTML Author's Guide (http://www.robotstxt.org/wc/exclusion-user.html) it says that robots.txt must be in the top level of a server's document space.

It also says that "if you rent space..." you usually aren't allowed to modify things at this level.

I and many others are using VHost or other approaches to host -- does this still apply?

C.

9:03 pm on Mar 13, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 28, 2002
posts:505
votes: 0


Generally, the robots.txt has to be in the document-root directory of the domain, which is defined in the web server's httpd.conf configuration file, and which is "/" from the internet web side's view, regardless if virtual or not. It is not tied to the root of the physical server's file system, so there is no problem with vhosts at all: just put the robots.txt file where the "/" of your domain is: if you have a "www.example.com/" site, simply put the robots.txt where it can be seen as "www.example.com/robots.txt".
On the opposite: if you have a site like "www.provider.com/private/my-dir/index.htm" where your own web site starts below some sub-directory, then you are lost, because your provider won't let you set a "www.provider.com/robots.txt" (and a "www.provider.com/private/my-dir/robots.txt" would be useless).

Regards,
R.

9:21 pm on Mar 13, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 7, 2003
posts:1048
votes: 0


Good - that is what I thought.

Any comment on when this changed?

I suppose you can't trust much of what you read these days, 'specially on the 'net.

Thanks!

9:46 pm on Mar 13, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 28, 2002
posts:505
votes: 0


Chewy,
... it had always been this way, and the original author described the same thing -- just with other and fewer words, leaving room for some uncertaincies.

Regards,
R.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members