A fairy tale;
Imagine for a moment, a meeting of the minds, one that creates a movement, a movement so strong that it unites the robots.txt creators of the universe. They would wield an incredible power, but only if they shared a collective creative inspiration. Powerful and large entities using content created by their followers were able to feed and grow on those offerings.
These behemoths grew to be like Gods, the pillars of the virtual world emerged as Hoogle, Cing, and Zahoo. Those who were smiled upon by the Gods would reap great riches, and the promise of that smile, the riches, kept them eagerly feeding their masters. But these masters were fickle Gods, with enormous power to smite, for when they grew angry, your offering would be refused. Your offering no longer palatable, and you no longer of use, after all there were many other offerings, so you could be made to lose contact with all around you, placed in a parallel world where very few can ever find you! No smiles, no riches, no ever growing log files. The Gods have spoken.
But these Gods had a weakness, Hoogle, Cing, and Zahoo were immensely competitive, and they answered to a greater power, the God of Robots, they were compelled to follow the scriptures as declared in the robots.txt, but what they did not know, is that the scriptures were written by those they grew to disdain.
Without warning, the collective creators of robots scripture seized the power that had always been theirs, and collectively determined which God would would be fed, and which would be starved. Being benevolent authors of robots scripture they did not wish to see the Gods' demise, but only to restore some appreciation of the offerings that had been taken for granted.
First they modified robots.txt to allow only Cing to accept the offerings, while Hoogle and Zahoo, wasted away. Hoogle and Zahoo asked what had they done wrong to be so treated, and the creators referred them to the using-other-people's-offerings-to-grow-your-business-with-grace-policy, perhaps a violation or transgression?. The creators then ran an update to the scriptures and allowed Hoogle to accept the offerings, denying Cing, and continuing to deny Zahoo.
The Gods became tense and nervous, they did not know what to do, there seemed to be not rhyme or reason to the scriptures, how could they know from one moment to the next whether they would meet their demise or prosper? Thus it came to be that a new deal was struck, the Gods recognizing the power of the creators and their own empty existence without them, and the creators recognizing the God's power to give and appreciate. Each needing the other to survive, a truly symbiotic relationship was formed. History however is not kind to those who ignore it, and so after a time ...