homepage Welcome to WebmasterWorld Guest from 54.166.84.82
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt
johnnstacy1

10+ Year Member



 
Msg#: 2 posted 12:59 am on Sep 1, 2003 (gmt 0)

If I have 50 sites operating on a server, do I need to place a robots.txt file in ever domain root folder or can I place it in one central location that can be used by all of the domains? Sorry for the newbie question.

 

PhraSEOlogy

10+ Year Member



 
Msg#: 2 posted 1:59 am on Sep 1, 2003 (gmt 0)

I think you have to put the file in the root of all the sites. Presumably each site is different and would probably want to disallow files/directories based upon their structure - unless each site is identical?

jamesa

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2 posted 4:26 am on Sep 1, 2003 (gmt 0)

You'd need to place it in the Document Root of each site, yes.

dmorison

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2 posted 8:19 am on Sep 1, 2003 (gmt 0)

If you just want to use robots.txt to ban particular user-agents (and therefore the same file would be ok for each site), then you can, if you're on a *nix box, maintain a single central copy; and create a soft link to it in each directory.

You still have to create all the soft links, but after that you can quickly make one change that will impact all sites.

The Linux command is "ln" to create a link, so if you were in the wwwroot directory of SiteA, you might type:

ln robots.txt /path/to/robots.txt

...where /path/to/robots.txt is your central copy.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved