To block or not to block (Archive.org)

Are you blocking this bot?

[mslina2002]

I just wanted to see what the practice is nowadays. I am not an expert and wanted to get some advice regarding this bot. I have all original content on my site and would like this to be protected (as much as possible) so would like to find out what most of you are doing regarding this.

I had it blocked for one time when I first created one of my site 2 years ago, however have unblocked the bot about 6 months now.

What are the advantages of blocking
* No stealing of content ?
* No duplicate content site?
* bot using too much resources ?

What are the disadvantages of blocking
* In case of copyright infringement you will have another way to prove the origin of your site ?

One more question, I have allowed the ia-archiver bot back in, but is the Nutch bot also being used for this?

[jdMorgan]

This really isn't an Apache-specific question, and you'll likely get more/better responses in our Webmaster General or Content, Writing and Copyright forums.

Internet Archiver is also a popular topic (or target) in our Search Engine Spider and User Agent Identification forum.

The bottom line is, "Does the benefit of the 'copyright protection" afforded by having your site in the Internet Archive outweigh the 'cost' of having that archive available to scrapers, copyright lawyers (not your own), and other people looking to make a buck on it?"

Only you can decide that, as you will get 20 different answers for every 20 Webmasters you ask about this.

Jim

[mslina2002]

That is a good question and that is probaby why I am flip flopping on this issue and seeing what else everyone is doing... looking at the pros here and just following along.

Can this post be moved by a mod here or should I repost in the appropriate forum.

Thanks for your help.