Welcome to WebmasterWorld Guest from 54.234.63.187

Forum Moderators: goodroi

Message Too Old, No Replies

where to place robots.txt?

robots newbie alert

     

Duke_of_Url

10:14 am on Jun 10, 2004 (gmt 0)

10+ Year Member



Hi

I've been reading a little about robots, as my bandwidth usage is going thru the roof.

My question is: where do I place the robots.txt file?

On my hosting package, I have folders set up, each of which contains the pages for a different site with the url pointing to that folder.
Do I just need one robots in the root above all these folders, or one robots.txt in each folder for each site?

TIA
DoU

Leosghost

10:31 am on Jun 10, 2004 (gmt 0)

WebmasterWorld Senior Member leosghost is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



On my hosting package, I have folders set up, each of which contains the pages for a different site with the url pointing to that folder.

what do you mean ..you have more than one url/domain on your hosting package? In which case you need a separate robots.txt for each url/domain. You need to explain what you've got set up a bit more clearly so as not to be steered the wrong way ..

bufferzone

10:47 am on Jun 10, 2004 (gmt 0)

10+ Year Member



In short, you need to place your robots.txt in the web root. This means that your robots.txt must be available through the URL [yoursite.com...] . If you have more than one domain, you must place one robots.txt in each domain, also virtual domains (more then one web site pr. server)

Duke_of_Url

11:35 am on Jun 10, 2004 (gmt 0)

10+ Year Member



Thanks all

From the root of my hosting package I have:

/folderABC/<all the files for website1>
/folderDEF/<all the files for website2>
..
..
and so on, with different urls pointing to each folder for each different site.

So I think I need to place a robots.txt in each of these folders, ie one per site? or if I place one robots in the root folder above all these, will that be enough?

TIA
DoU

isitreal

5:11 pm on Jun 10, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



No, the robots.txt file is a file that must be available to site visitors like search spiders, that means it has to be at the root level folder of each and every website, not one level above that, you can't access a file through the web if it's one level above the root folder /, if you can't type in yoursite.com/robots.txt and then view your robots.txt file, nothing else can see it either over the web.

Duke_of_Url

10:17 pm on Jun 10, 2004 (gmt 0)

10+ Year Member



thanks all, will have a try tomorrow with it!

much appreciated
DoU

 

Featured Threads

Hot Threads This Week

Hot Threads This Month