Welcome to WebmasterWorld Guest from 54.242.231.210

Forum Moderators: incrediBILL & lawman

Message Too Old, No Replies

Lowly robots.txt wields a hidden power

Determine the fate of search through collective robotting

     

Air

5:17 pm on Jan 2, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 10, 2000
posts:1253
votes: 0


A fairy tale;

Imagine for a moment, a meeting of the minds, one that creates a movement, a movement so strong that it unites the robots.txt creators of the universe. They would wield an incredible power, but only if they shared a collective creative inspiration. Powerful and large entities using content created by their followers were able to feed and grow on those offerings.

These behemoths grew to be like Gods, the pillars of the virtual world emerged as Hoogle, Cing, and Zahoo. Those who were smiled upon by the Gods would reap great riches, and the promise of that smile, the riches, kept them eagerly feeding their masters. But these masters were fickle Gods, with enormous power to smite, for when they grew angry, your offering would be refused. Your offering no longer palatable, and you no longer of use, after all there were many other offerings, so you could be made to lose contact with all around you, placed in a parallel world where very few can ever find you! No smiles, no riches, no ever growing log files. The Gods have spoken.

But these Gods had a weakness, Hoogle, Cing, and Zahoo were immensely competitive, and they answered to a greater power, the God of Robots, they were compelled to follow the scriptures as declared in the robots.txt, but what they did not know, is that the scriptures were written by those they grew to disdain.

Without warning, the collective creators of robots scripture seized the power that had always been theirs, and collectively determined which God would would be fed, and which would be starved. Being benevolent authors of robots scripture they did not wish to see the Gods' demise, but only to restore some appreciation of the offerings that had been taken for granted.

First they modified robots.txt to allow only Cing to accept the offerings, while Hoogle and Zahoo, wasted away. Hoogle and Zahoo asked what had they done wrong to be so treated, and the creators referred them to the using-other-people's-offerings-to-grow-your-business-with-grace-policy, perhaps a violation or transgression?. The creators then ran an update to the scriptures and allowed Hoogle to accept the offerings, denying Cing, and continuing to deny Zahoo.

The Gods became tense and nervous, they did not know what to do, there seemed to be not rhyme or reason to the scriptures, how could they know from one moment to the next whether they would meet their demise or prosper? Thus it came to be that a new deal was struck, the Gods recognizing the power of the creators and their own empty existence without them, and the creators recognizing the God's power to give and appreciate. Each needing the other to survive, a truly symbiotic relationship was formed. History however is not kind to those who ignore it, and so after a time ...

6:01 pm on Jan 7, 2010 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts: 38057
votes: 12


an intresting tale Air. I wish...I wish we had that power to unite.
7:57 pm on Apr 1, 2010 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 8, 2005
posts:147
votes: 0


History however is not kind to those who ignore it, and so after a time ...

The Hoogle, Cing, and Zahoo Gods got powerful again, driven by the self preservation excess, desired to be creators them self. They slowly pushed the robots.txt creators from the fertile and valuable lands and build their own portals, halls and forums. They again imposed the new set of laws and high levies and begin to banish creators as before for minor omissions, or lack of gold to the darkness of quest abysses to wither and vanish.

Seeing that they have to fight or perish, the creators of robots scripture went to the wise men of Spider Code for help. The wise men advised the creators to build FireHounds and give them to the all common folks for free, so they can find and see the all creator’s robots.txt vast domains …
9:05 pm on Apr 1, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 12, 2006
posts:2533
votes: 36


there's no actual law that says a search engine is bound to obey robots.txt (as far as i know). so if a business as big as bong, zahoo and boogle was faced with utter ruin, it's a no-brainer what they'd do. they'd just ignore it.
9:16 pm on Apr 1, 2010 (gmt 0)

Moderator from US 

WebmasterWorld Administrator lifeinasia is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 10, 2005
posts:5590
votes: 27


they'd just ignore it.

Or they'd be quickly defeated by the demi-gods (anti-gods?) who didn't obey robots.txt.
9:40 pm on Apr 1, 2010 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator mack is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 15, 2001
posts:7564
votes: 4


If they fought against the power of robots scripture they would be banished to the realms of htaccess.

Mack.
2:18 am on Apr 2, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1743
votes: 40


And then there was the last of levels of defense; it was the mighty anti-gods who though of it way earlier, and in advance. The Walls of Fire, soft design, in secrete Code that spread its wings all over universe in altered-cryptic forms. The name was given also: PAYWALLS.

don't ask, long day...
4:54 am on Apr 2, 2010 (gmt 0)

Senior Member from LK 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Nov 16, 2005
posts:2477
votes: 30


they'd just ignore it.

There may be any actual law, but it is of some importance in copyright cases because it is an accepted industry standard - in particular it provides them with a defence.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members