Forum Moderators: open
User-agent: *
Disallow: /folder/
Disallow: /page.htm/
Disallow: /another_page.htm/
On checking Googles own robots.txt file, it reads as follows (I haven't used the entire file, just a part of it)
User-agent: *
Disallow: /search
Disallow: /groups
Disallow: /images
Disallow: /catalogs
Disallow: /catalog_list
Disallow: /news
Disallow: /pagead/
etc, etc, etc, etc, more stuff etc........
Disallow: /microsoft?
Disallow: /unclesam?
I notice the lack of trailing / on most (but not all) entries. Which is the correct format I wonder?