homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How is correct code within robots.txt?

 9:02 am on Nov 28, 2008 (gmt 0)

I would like to place correct code to block search engines to see content:
User-agent: *
Disallow: /subdomain.mydomain.com/
Disallow: /upload/mydomain/

Is this 100% correct code or there is possible to validate robots.txt?
Need help for this simple question.Thank you



 1:44 pm on Nov 28, 2008 (gmt 0)

No one could possibly know if the code was correct, because we don't know the structure of your site, nor do we know all the URLs that you want to allow and all the URLs that you want to block.

Google WebmasterTools can give you some reports that might be helpful - but it is down to you to interpret what they say.

However, you must put the file in the root of the domain that it needs to apply to, the disallow URLs must contain paths and filenames and must NOT include domain names (unless that's actually the name of a folder).

You might not need to end the disallow statement with the trailing slash, because having the trailing slash in the disallow means that /folder isn't blocked. That may, or may not, be a problem.


 1:50 pm on Nov 28, 2008 (gmt 0)

if I replace subdomain value and I create folders. will it work?


 11:07 am on Nov 29, 2008 (gmt 0)

if you have a google webmaster tools account you can test your robots.txt file [google.com] there.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved