homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Compressing Robots.txt
Is this robots.txt compression valid?

5+ Year Member

Msg#: 3914556 posted 5:24 pm on May 15, 2009 (gmt 0)

I wanted to compress my robots.txt and I came to this below:

User-agent: Googlebot
User-agent: Googlebot-Mobile
Noindex: /*?
Disallow: /*%23
Disallow: *bots=nocrawl
Noindex: *bots=nocrawl
Noindex: /*~r
Noindex: *bots=noindex
Noindex: /includes/

User-agent: AdsBot-Google
User-agent: Mediapartners-Google
Disallow: /

User-agent: Googlebot-Image
Disallow: /images/
Allow: /*.jpg
Allow: /*.jpeg
Allow: /*.png
Allow: /*.gif

User-agent: WDG_SiteValidator
Disallow: /neuterale.php

User-Agent: MJ12bot

User-agent: Slurp
Crawl-delay: 300
User-agent: Msnbot
Crawl-delay: 120
User-agent: Teoma
Crawl-delay: 240
User-agent: *
Disallow: /*?
Disallow: /*%23
Disallow: /*~r
Disallow: *bots=nocrawl
Disallow: *bots=noindex
Disallow: /includes/

Is that valid? Will it work? If not, what do you see wrong?
About the "noindex" directive no worries. It is unofficially supported by Googlebot.

Thanks in advance for your kind support,




WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3914556 posted 12:46 pm on May 18, 2009 (gmt 0)

It might work but I think you are probably going to have problems with at least one bot.

I don't understand why you are trying to compress your robots.txt file? It seems you are trying too hard to save a few kb.

I would worry more about making sure the robots.txt is accessible & understandable to search bots. Search robots can get easily confused. Your robots.txt should be formatted exactly as the search engines request if you want to make sure the robots follow your instructions.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved