homepage Welcome to WebmasterWorld Guest from 54.147.196.159
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt
nervous beginner seeks reassurance
jgar

10+ Year Member



 
Msg#: 133 posted 5:34 am on Aug 11, 2002 (gmt 0)

I have read some messages here about the robots.txt file, and understand it is vital to syntax it correctly.

Being a newcomer to the use of the robots.txt file, can someone please reassure me that the following is correct:

User-agent: *
Disallow: /Application/Pdcd/Langlab/Changelanggen.asp

The aim is to stop Googlebot trying to look at the page Changelanggen.asp, and the text file robots.txt is placed in the root directory.

Appreciate any comments.

 

pageoneresults

WebmasterWorld Senior Member pageoneresults us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 133 posted 6:04 am on Aug 11, 2002 (gmt 0)

Hello jgar, you said...

Disallow: /Application/Pdcd/Langlab/Changelanggen.asp

Yes, that is correct. All other files within that directory will be indexed. Root directory is the only place you can put your robots.txt file.

shelleycat

10+ Year Member



 
Msg#: 133 posted 7:06 am on Aug 11, 2002 (gmt 0)

I'm also new to using robots.txt and I found the validator provided by webmasterworld greatly helpful. Take a look here [searchengineworld.com...] :)

(darn typos)

jgar

10+ Year Member



 
Msg#: 133 posted 7:30 am on Aug 11, 2002 (gmt 0)

Thanks for the tips - the validator is a great resource

ciml

WebmasterWorld Senior Member ciml us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 133 posted 4:51 pm on Aug 12, 2002 (gmt 0)

Just one extra comment.

robots.txt is case sensitive, so "Disallow: /Application" disallows anything beginning with "/Application" but not "/application".

Has anyone checked that Google does this right?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved