Welcome to WebmasterWorld Guest from 50.17.5.36

Forum Moderators: incrediBILL & lawman

Message Too Old, No Replies

Can you possibly sue someone for . . .

not honouring the robots.txt while crawling?

     
9:53 am on Apr 24, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2004
posts:1867
votes: 0


Can you possibly sue someone for not honouring the robots.txt while crawling?

Can you?

I do understand the technical difficulties involved in identifying the crawler.
More interested in the law side than the possible other issues it may arise.

[edited by: Habtom at 9:56 am (utc) on April 24, 2008]

11:36 am on Apr 24, 2008 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 4, 2001
posts: 2143
votes: 7


Probably could. In the US, you can sue anyone for anything, but that doesn't mean you'll win (I'm guessing you're in the UK from the way you spelled "honour"). The safest answer though is talk to a lawyer, preferably one that specializes in internet issues.

Marshall

11:48 am on Apr 24, 2008 (gmt 0)

Junior Member

5+ Year Member

joined:May 25, 2006
posts: 182
votes: 0


I don't think so.

The robots.txt RFC [robotstxt.org] says in point 2:

[...] The technique specified in this memo allows Web site administrators to indicate to visiting robots which parts of the site should be avoided. It is solely up to the visiting robot to consult this information and act accordingly. Blocking parts of the Web site regardless of a robot's compliance with this method are outside the scope of this memo. [...]

IIRC the RFCs are the de facto 'laws' of the internet defining protocols and stuff like that. And as long as it says 'It is solely up to the visiting robot to consult this information and act accordingly' you can't sue anybody for ignoring robots.txt.

7:32 pm on Apr 24, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:May 27, 2003
posts:503
votes: 0


> (I'm guessing you're in the UK from the way you spelled "honour").

Or maybe Australia, Belize, Canada, Denmark, Ethiopia, Finland, Guyana, Honduras, India, Jamaica, Kuwait, Latvia or Maldives, as examples from the first half of the alphabet. In Hatbom's case, the United Arab Emirates.

Now admittedly, English is not the primary language of a number of countries I mentioned, but when the government speaks in English, they do say "honour" or "colour" or "neighbour" or...


Has anyone even tried suing Google, MSN or Yahoo when their 'bots have disobeyed robots.txt?

4:04 am on Apr 27, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2004
posts:1867
votes: 0


It is solely up to the visiting robot to consult this information and act accordingly.

It is optional then. It probably could have been better for the site owner to set restrictions to his/her own site using whatever method which should be respected by everybody else.

balam, I have lived in the UAE for the last 3 years now hopefully this being my last.