homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

nervous beginner seeks reassurance

 5:34 am on Aug 11, 2002 (gmt 0)

I have read some messages here about the robots.txt file, and understand it is vital to syntax it correctly.

Being a newcomer to the use of the robots.txt file, can someone please reassure me that the following is correct:

User-agent: *
Disallow: /Application/Pdcd/Langlab/Changelanggen.asp

The aim is to stop Googlebot trying to look at the page Changelanggen.asp, and the text file robots.txt is placed in the root directory.

Appreciate any comments.



 6:04 am on Aug 11, 2002 (gmt 0)

Hello jgar, you said...

Disallow: /Application/Pdcd/Langlab/Changelanggen.asp

Yes, that is correct. All other files within that directory will be indexed. Root directory is the only place you can put your robots.txt file.


 7:06 am on Aug 11, 2002 (gmt 0)

I'm also new to using robots.txt and I found the validator provided by webmasterworld greatly helpful. Take a look here [searchengineworld.com...] :)

(darn typos)


 7:30 am on Aug 11, 2002 (gmt 0)

Thanks for the tips - the validator is a great resource


 4:51 pm on Aug 12, 2002 (gmt 0)

Just one extra comment.

robots.txt is case sensitive, so "Disallow: /Application" disallows anything beginning with "/Application" but not "/application".

Has anyone checked that Google does this right?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved