homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Can IMDB do this?
charge per unwanted request

 9:46 pm on Apr 19, 2004 (gmt 0)

I was looking at imdb.com's robots.txt, and noticed at the bottom:

# Robots that cause a denial of service will be charged $0.01 per unwanted
# request to this site. Follow the guidelines + robots.txt convention and
# we can all live happily together.

Can they legally charge someone $0.01 per unwanted request, or is this just more of a scare tactic?



 9:51 pm on Apr 19, 2004 (gmt 0)

They can do anything they want. But collecting the money might be a completely different problem.


 10:19 pm on Apr 19, 2004 (gmt 0)

Jrap, that's pretty funny, I haven't seen it before. I guess if they sue someone, they can throw this provision in along with the kitchen sink when they tally up damages.

It seems like robots.txt has occasionally been shown as valid by courts, at least as far as determining whether someone had the right to spider another site. Since robots can't read the amount they are supposed to pay, though, I think the site owner would have a lot of difficulty proving that an enforceable contract exists.

"Unwanted" is kind of nebulous, too, as is the exact meaning of "denial of service". On the other hand, if they were getting persistent abuse by a robot and they served written notice to the abuser, it might be a more interesting day in court.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved