I wanted to compress my robots.txt and I came to this below: User-agent: Googlebot
User-agent: Googlebot-Mobile
Noindex: /*?
Disallow: /*%23
Disallow: *bots=nocrawl
Noindex: *bots=nocrawl
Noindex: /*~r
Noindex: *bots=noindex
Noindex: /includes/
User-agent: AdsBot-Google
User-agent: Mediapartners-Google
Disallow: /
User-agent: Googlebot-Image
Disallow: /images/
Allow: /*.jpg
Allow: /*.jpeg
Allow: /*.png
Allow: /*.gif
User-agent: WDG_SiteValidator
Disallow: /neuterale.php
User-Agent: MJ12bot
Disallow:
User-agent: Slurp
Crawl-delay: 300
User-agent: Msnbot
Crawl-delay: 120
User-agent: Teoma
Crawl-delay: 240
User-agent: *
Disallow: /*?
Disallow: /*%23
Disallow: /*~r
Disallow: *bots=nocrawl
Disallow: *bots=noindex
Disallow: /includes/
Is that valid? Will it work? If not, what do you see wrong?
About the "noindex" directive no worries. It is unofficially supported by Googlebot.
Thanks in advance for your kind support,
John
