Welcome to WebmasterWorld Guest from

Forum Moderators: not2easy

Message Too Old, No Replies

Linking to News Sites.

...articles disappear.



3:02 pm on May 2, 2007 (gmt 0)

10+ Year Member

I have a news aggregate site that posts links to articles and then allows user comments (in a Slashdot-style) with a niche topic.

However, it seems the articles that I link to tend to disappear after a few weeks or so. Any idea why that may be?

I was wondering if I might be able to cache the articles, similar to the way Archive.org does (assuming the site doesn't forbid it with robots.txt) and link to the cache so I know that I'm always linking to a working version of the article.


1:51 am on May 3, 2007 (gmt 0)

10+ Year Member

The articles get taken down on AP partner news sites generally after 14 days by contract. Only AHN lets 180 day archiving I believe

Altough you can cache (copy) their content you probably shouldn't.

1: You generally violate copyrignt when you cache.. Are you an AHN, AP, Reuters, AFP licensee? Get permission... News organizations register their copyrights, that means statutory damages.

2: If the article gets changed or retracted and then you fail to update then you can be liable for libel or even worse. Lawyers, specifically lawyers for celebrities live for this.


7:18 am on May 3, 2007 (gmt 0)

10+ Year Member

So how does something like the Wayback Machine get away with caching the entire internet? Certainly they obey robots.txt, and I would think that would be the only criteria.

Or is this one of those "I'd probably be right if I followed robots.txt, but I could get dragged through court by lawyers and it wouldn't be worth it" sort of situations?


11:14 am on May 3, 2007 (gmt 0)

10+ Year Member

The Internet Archive at archive.org (aka 'wayback machine') is entirely non-commercial, however they still get sued occasionally.

You can almost guarantee a suit if your site is in anyway commercial in nature or seeks any revenues.

You should read the FAQ and news section.


Featured Threads

Hot Threads This Week

Hot Threads This Month