homepage Welcome to WebmasterWorld Guest from 54.237.98.229
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
where to place robots.txt?
robots newbie alert
Duke_of_Url

10+ Year Member



 
Msg#: 403 posted 10:14 am on Jun 10, 2004 (gmt 0)

Hi

I've been reading a little about robots, as my bandwidth usage is going thru the roof.

My question is: where do I place the robots.txt file?

On my hosting package, I have folders set up, each of which contains the pages for a different site with the url pointing to that folder.
Do I just need one robots in the root above all these folders, or one robots.txt in each folder for each site?

TIA
DoU

 

Leosghost

WebmasterWorld Senior Member leosghost us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 403 posted 10:31 am on Jun 10, 2004 (gmt 0)

On my hosting package, I have folders set up, each of which contains the pages for a different site with the url pointing to that folder.

what do you mean ..you have more than one url/domain on your hosting package? In which case you need a separate robots.txt for each url/domain. You need to explain what you've got set up a bit more clearly so as not to be steered the wrong way ..

bufferzone

10+ Year Member



 
Msg#: 403 posted 10:47 am on Jun 10, 2004 (gmt 0)

In short, you need to place your robots.txt in the web root. This means that your robots.txt must be available through the URL [yoursite.com...] . If you have more than one domain, you must place one robots.txt in each domain, also virtual domains (more then one web site pr. server)

Duke_of_Url

10+ Year Member



 
Msg#: 403 posted 11:35 am on Jun 10, 2004 (gmt 0)

Thanks all

From the root of my hosting package I have:

/folderABC/<all the files for website1>
/folderDEF/<all the files for website2>
..
..
and so on, with different urls pointing to each folder for each different site.

So I think I need to place a robots.txt in each of these folders, ie one per site? or if I place one robots in the root folder above all these, will that be enough?

TIA
DoU

isitreal

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 403 posted 5:11 pm on Jun 10, 2004 (gmt 0)

No, the robots.txt file is a file that must be available to site visitors like search spiders, that means it has to be at the root level folder of each and every website, not one level above that, you can't access a file through the web if it's one level above the root folder /, if you can't type in yoursite.com/robots.txt and then view your robots.txt file, nothing else can see it either over the web.

Duke_of_Url

10+ Year Member



 
Msg#: 403 posted 10:17 pm on Jun 10, 2004 (gmt 0)

thanks all, will have a try tomorrow with it!

much appreciated
DoU

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved