homepage Welcome to WebmasterWorld Guest from 54.243.13.30
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Visit PubCon.com
Home / Forums Index / Local / Foo
Forum Library, Charter, Moderators: incrediBILL & lawman

Foo Forum

    
Can you possibly sue someone for . . .
not honouring the robots.txt while crawling?
Habtom




msg:3633965
 9:53 am on Apr 24, 2008 (gmt 0)

Can you possibly sue someone for not honouring the robots.txt while crawling?

Can you?

I do understand the technical difficulties involved in identifying the crawler.
More interested in the law side than the possible other issues it may arise.

[edited by: Habtom at 9:56 am (utc) on April 24, 2008]

 

Marshall




msg:3634035
 11:36 am on Apr 24, 2008 (gmt 0)

Probably could. In the US, you can sue anyone for anything, but that doesn't mean you'll win (I'm guessing you're in the UK from the way you spelled "honour"). The safest answer though is talk to a lawyer, preferably one that specializes in internet issues.

Marshall

OutdoorWebcams




msg:3634043
 11:48 am on Apr 24, 2008 (gmt 0)

I don't think so.

The robots.txt RFC [robotstxt.org] says in point 2:

[...] The technique specified in this memo allows Web site administrators to indicate to visiting robots which parts of the site should be avoided. It is solely up to the visiting robot to consult this information and act accordingly. Blocking parts of the Web site regardless of a robot's compliance with this method are outside the scope of this memo. [...]

IIRC the RFCs are the de facto 'laws' of the internet defining protocols and stuff like that. And as long as it says 'It is solely up to the visiting robot to consult this information and act accordingly' you can't sue anybody for ignoring robots.txt.

balam




msg:3634429
 7:32 pm on Apr 24, 2008 (gmt 0)

> (I'm guessing you're in the UK from the way you spelled "honour").

Or maybe Australia, Belize, Canada, Denmark, Ethiopia, Finland, Guyana, Honduras, India, Jamaica, Kuwait, Latvia or Maldives, as examples from the first half of the alphabet. In Hatbom's case, the United Arab Emirates.

Now admittedly, English is not the primary language of a number of countries I mentioned, but when the government speaks in English, they do say "honour" or "colour" or "neighbour" or...


Has anyone even tried suing Google, MSN or Yahoo when their 'bots have disobeyed robots.txt?

Habtom




msg:3636024
 4:04 am on Apr 27, 2008 (gmt 0)

It is solely up to the visiting robot to consult this information and act accordingly.

It is optional then. It probably could have been better for the site owner to set restrictions to his/her own site using whatever method which should be respected by everybody else.

balam, I have lived in the UAE for the last 3 years now hopefully this being my last.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Local / Foo
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved