homepage Welcome to WebmasterWorld Guest from 23.23.2.137
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How is correct code within robots.txt?
toplisek




msg:3795871
 9:02 am on Nov 28, 2008 (gmt 0)

I would like to place correct code to block search engines to see content:
User-agent: *
Disallow: /subdomain.mydomain.com/
Disallow: /upload/mydomain/

Is this 100% correct code or there is possible to validate robots.txt?
Need help for this simple question.Thank you

 

g1smd




msg:3795984
 1:44 pm on Nov 28, 2008 (gmt 0)

No one could possibly know if the code was correct, because we don't know the structure of your site, nor do we know all the URLs that you want to allow and all the URLs that you want to block.

Google WebmasterTools can give you some reports that might be helpful - but it is down to you to interpret what they say.

However, you must put the file in the root of the domain that it needs to apply to, the disallow URLs must contain paths and filenames and must NOT include domain names (unless that's actually the name of a folder).

You might not need to end the disallow statement with the trailing slash, because having the trailing slash in the disallow means that /folder isn't blocked. That may, or may not, be a problem.

toplisek




msg:3795986
 1:50 pm on Nov 28, 2008 (gmt 0)

Hi,
if I replace subdomain value and I create folders. will it work?

phranque




msg:3796525
 11:07 am on Nov 29, 2008 (gmt 0)

if you have a google webmaster tools account you can test your robots.txt file [google.com] there.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved