Welcome to WebmasterWorld Guest from 23.20.6.20

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt

nervous beginner seeks reassurance

     
5:34 am on Aug 11, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 18, 2002
posts:101
votes: 0


I have read some messages here about the robots.txt file, and understand it is vital to syntax it correctly.

Being a newcomer to the use of the robots.txt file, can someone please reassure me that the following is correct:

User-agent: *
Disallow: /Application/Pdcd/Langlab/Changelanggen.asp

The aim is to stop Googlebot trying to look at the page Changelanggen.asp, and the text file robots.txt is placed in the root directory.

Appreciate any comments.

6:04 am on Aug 11, 2002 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts:12169
votes: 55


Hello jgar, you said...

Disallow: /Application/Pdcd/Langlab/Changelanggen.asp

Yes, that is correct. All other files within that directory will be indexed. Root directory is the only place you can put your robots.txt file.

7:06 am on Aug 11, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:July 26, 2002
posts:166
votes: 0


I'm also new to using robots.txt and I found the validator provided by webmasterworld greatly helpful. Take a look here [searchengineworld.com...] :)

(darn typos)

7:30 am on Aug 11, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 18, 2002
posts:101
votes: 0


Thanks for the tips - the validator is a great resource
4:51 pm on Aug 12, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 22, 2001
posts:3805
votes: 2


Just one extra comment.

robots.txt is case sensitive, so "Disallow: /Application" disallows anything beginning with "/Application" but not "/application".

Has anyone checked that Google does this right?