Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Want to Deny Google from Indexing, Crawling My Private Site

erlandc - 4:53 pm on Aug 11, 2010 (gmt 0)


Is this correct? Google keeps "GET /audio/whatever.ext HTTP/1.1" How do I stop all access? Do I have to resort to .htaccess? It's a Private Site. Thanks. E

User-agent: Googlebot
Disallow: *googlebot=nocrawl$
User-agent: Googlebot-Image
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: *
Disallow: /*.jpg
Disallow: /*.gif
# go away
User-agent: *
Disallow: /
Disallow: /
Disallow: /http://www.mysite.tv/
Disallow: /otherpages.html

Thread source:: http://www.webmasterworld.com/robots_txt/4185458.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com