If what you want to do is stop Google from indexing all pages on your site, simply use this directive in robots.txt: User-agent: Googlebot
Disallow: / If you want Googlebot to crawl some pages but not others, say to restrict it from one particular directory, use this directive: User-agent: Googlebot
Disallow: /directorytorestrict/ Basic information about robots.txt can be found at robotstxt.org. Specific information on how Google has implemented robots.txt can be found here.
| 
