homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Need assistance with robot controls
need help with a robot .txt file

10+ Year Member

Msg#: 477 posted 1:44 am on Oct 25, 2004 (gmt 0)

I am having a difficult time with robots. While checking stats I found that MSN was trying to reach subdomain pages through my main domains.

That is, I found MSN spidering
[domain.com...] (a path that does not exist)

It should be spidering [subdomain.com...]

(By 'subdomain' I mean shared hosting. 'www.domain.com' and 'www.subdomain.com' come up as live sites, not 'www.subdomain.domain.com').

I have been advised to "put a robot.txt file in the route that excludes the sub directories of the subdomains. Then put a robot.txt inside the subdomain dir that excludes it from going back to the route" but I have no clue as to how to do this.

Any help is greatly appreciated!



WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 477 posted 11:31 pm on Oct 29, 2004 (gmt 0)

Do you map subdomains to subdirectories or anything like that? -- An error in implementing such a mapping function could confuse the 'bot.

You might also want to implement a 301-Moved Permanently redirect to redirect all requests for www.<subdomain>.domain.com to subdomain.domain.com in order to get the search listings corrected sooner, but look for the underlying problem first.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved