How to disallow widgets from being indexed

Order of User-agents

[albatros]

I wonder whether the order of User-agent: * and User-agent: Googlebot sections matters. Is there a difference between the way Googlebot handles these robots.txt files?

User-agent: *
Disallow: /cgi-bin/
User-agent: Googlebot
Disallow: /widgets

User-agent: Googlebot
Disallow: /widgets
User-agent: *
Disallow: /cgi-bin/

[Pfui]

I've found that with Googlebot/googlebot.com, order doesn't matter because that bot looks for its name (so to speak).

But if you also have a blanket disallow somewhere in your list...

User-agent: *
Disallow: /

...older bots for which you have specific instructions may stop after reading it.

So if you don't want to block all (robots.txt-respecting) bots, and you use a blanket disallow, put your named bots with their instructions first.

See Also: The Web Robots FAQ [robotstxt.org]