homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Local / Foo
Forum Library, Charter, Moderators: incrediBILL & lawman

Foo Forum

Lowly robots.txt wields a hidden power
Determine the fate of search through collective robotting

WebmasterWorld Senior Member 10+ Year Member

Msg#: 4052941 posted 5:17 pm on Jan 2, 2010 (gmt 0)

A fairy tale;

Imagine for a moment, a meeting of the minds, one that creates a movement, a movement so strong that it unites the robots.txt creators of the universe. They would wield an incredible power, but only if they shared a collective creative inspiration. Powerful and large entities using content created by their followers were able to feed and grow on those offerings.

These behemoths grew to be like Gods, the pillars of the virtual world emerged as Hoogle, Cing, and Zahoo. Those who were smiled upon by the Gods would reap great riches, and the promise of that smile, the riches, kept them eagerly feeding their masters. But these masters were fickle Gods, with enormous power to smite, for when they grew angry, your offering would be refused. Your offering no longer palatable, and you no longer of use, after all there were many other offerings, so you could be made to lose contact with all around you, placed in a parallel world where very few can ever find you! No smiles, no riches, no ever growing log files. The Gods have spoken.

But these Gods had a weakness, Hoogle, Cing, and Zahoo were immensely competitive, and they answered to a greater power, the God of Robots, they were compelled to follow the scriptures as declared in the robots.txt, but what they did not know, is that the scriptures were written by those they grew to disdain.

Without warning, the collective creators of robots scripture seized the power that had always been theirs, and collectively determined which God would would be fed, and which would be starved. Being benevolent authors of robots scripture they did not wish to see the Gods' demise, but only to restore some appreciation of the offerings that had been taken for granted.

First they modified robots.txt to allow only Cing to accept the offerings, while Hoogle and Zahoo, wasted away. Hoogle and Zahoo asked what had they done wrong to be so treated, and the creators referred them to the using-other-people's-offerings-to-grow-your-business-with-grace-policy, perhaps a violation or transgression?. The creators then ran an update to the scriptures and allowed Hoogle to accept the offerings, denying Cing, and continuing to deny Zahoo.

The Gods became tense and nervous, they did not know what to do, there seemed to be not rhyme or reason to the scriptures, how could they know from one moment to the next whether they would meet their demise or prosper? Thus it came to be that a new deal was struck, the Gods recognizing the power of the creators and their own empty existence without them, and the creators recognizing the God's power to give and appreciate. Each needing the other to survive, a truly symbiotic relationship was formed. History however is not kind to those who ignore it, and so after a time ...



WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 4052941 posted 6:01 pm on Jan 7, 2010 (gmt 0)

an intresting tale Air. I wish...I wish we had that power to unite.


5+ Year Member

Msg#: 4052941 posted 7:57 pm on Apr 1, 2010 (gmt 0)

History however is not kind to those who ignore it, and so after a time ...

The Hoogle, Cing, and Zahoo Gods got powerful again, driven by the self preservation excess, desired to be creators them self. They slowly pushed the robots.txt creators from the fertile and valuable lands and build their own portals, halls and forums. They again imposed the new set of laws and high levies and begin to banish creators as before for minor omissions, or lack of gold to the darkness of quest abysses to wither and vanish.

Seeing that they have to fight or perish, the creators of robots scripture went to the wise men of Spider Code for help. The wise men advised the creators to build FireHounds and give them to the all common folks for free, so they can find and see the all creator’s robots.txt vast domains …


WebmasterWorld Senior Member 5+ Year Member

Msg#: 4052941 posted 9:05 pm on Apr 1, 2010 (gmt 0)

there's no actual law that says a search engine is bound to obey robots.txt (as far as i know). so if a business as big as bong, zahoo and boogle was faced with utter ruin, it's a no-brainer what they'd do. they'd just ignore it.


WebmasterWorld Administrator lifeinasia us a WebmasterWorld Top Contributor of All Time 5+ Year Member

Msg#: 4052941 posted 9:16 pm on Apr 1, 2010 (gmt 0)

they'd just ignore it.

Or they'd be quickly defeated by the demi-gods (anti-gods?) who didn't obey robots.txt.


WebmasterWorld Administrator mack us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 4052941 posted 9:40 pm on Apr 1, 2010 (gmt 0)

If they fought against the power of robots scripture they would be banished to the realms of htaccess.



WebmasterWorld Senior Member 5+ Year Member

Msg#: 4052941 posted 2:18 am on Apr 2, 2010 (gmt 0)

And then there was the last of levels of defense; it was the mighty anti-gods who though of it way earlier, and in advance. The Walls of Fire, soft design, in secrete Code that spread its wings all over universe in altered-cryptic forms. The name was given also: PAYWALLS.

don't ask, long day...


WebmasterWorld Senior Member 5+ Year Member

Msg#: 4052941 posted 4:54 am on Apr 2, 2010 (gmt 0)

they'd just ignore it.

There may be any actual law, but it is of some importance in copyright cases because it is an accepted industry standard - in particular it provides them with a defence.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Local / Foo
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved