seems dead links may be a new factor in recent algo changes - what is best tool for checking for these
3:00 am on May 24, 2012 (gmt 0)
We rarely discuss SEO tools because of the promotional risks involved. But this is a valuable question and worthy of discussion. I've seen checkers that do a good job technically but have a terrible interface, for instance. And vice versa.
So let's share what we found is good to fight link rot - our best backlink checkers.
3:19 am on May 24, 2012 (gmt 0)
LinkScrubber by our own Incredibill and/or Screaming Frog Spider will hook you up just right.
11:37 am on May 24, 2012 (gmt 0)
i've tried an early (alpha? pre-alpha?) version of linkscrubber and based on what a saw then it would probably give you the most comprehensive results at this point. detection of 200 OK parked pages, for example. i recently tried screaming frog on metmeg's advice but not enough experience to comment on it yet. i've used xenu link sleuth extensively and it will quickly report on 404 Not Found, etc for outbound links, but you'll miss soft 404s unless you can manually identify error page titles, for example. i will sometimes just use shell programming and a list of urls to look at responses and custom filter things based on the situation at hand. in this case i will use a command line user agent such as lwp-request (or wget or curl or whatever you prefer). note that in some cases you'll have your access blocked using some of these user agents.
11:42 am on May 24, 2012 (gmt 0)
I'm wondering why Google doesn't report bad links from your site too you in webmaster tools. Seems like it would be an easy feature for them to implement. They certainly know which sites you link to. They also know the status of pretty much every url on the internet. They could even go further and report the content of page that you link to has changed substantially.
Google: if you are listening, please add a broken links service to webmaster tools.
3:54 pm on May 24, 2012 (gmt 0)
That would be huge if Google added it to WHM. We should get louder about them adding that feature.
Until they do, I have been using Web Link Validator from REL software for about 5 years now. It is awesome and creates a quick report for you.
5:17 pm on May 24, 2012 (gmt 0)
LinkScrubber is particularly good, because it checks for so much more than just a "yes or no". Here's a partial list of what it tries to identify on the other end of your link:
Expired Registration Redemption Pages
Default Control Panel Pages
Default Server Pages
Pharma Hacked Pages
Injection Exploit Scripts
5:41 pm on May 24, 2012 (gmt 0)
I'm wondering why Google doesn't report bad links from your site too you in webmaster tools.
Why do you think that there is a penalty? Surely the keepers of the crypt aren't that simple, otherwise that would mean that the older the site the greater the penalty because in the course of 10-12 years the site could have been rebuilt several times. Simply upgrading through, html to cgi, php or ASP and then Asp.net or Wordpress would leave numerous broken links each time.
I know of one site that was created in 1998 that increased its content by about 600 pages each year, and then in 2008 was hijacked by monetisers who ruined its credibility before selling it back. In the meantime they continued under a new domain name, but found that domain age counted for too much so began using the original name again. All content was recently changed and most pages now use a different file extension, so you can imagine how many broken links that will leave out there.
In fact it has probably got more broken backlinks against it than good ones, but it still appears between 1-5 for its main keywords.
8:13 pm on May 24, 2012 (gmt 0)
When I'm on a cleanup campaign I find it's productive to run more than one link checking tool, because they have different strengths.
One that hasn't been mentioned yet is Integrity which is Mac-friendly.
8:18 pm on May 24, 2012 (gmt 0)
i just noticed "backlink" in the title which is really a different problem than dead urls you are linking out to.
10:41 pm on May 24, 2012 (gmt 0)
Whoah -that is a bit confusing. If a backlink is dead, you don't need to do anything about it. Or if you want to clean up bad backlinks, we've got a whole thread about that process [webmasterworld.com]
11:10 pm on May 24, 2012 (gmt 0)
I find dead backlinks by hanging a logging script off the 404 page and measuring all accesses to it.
3:51 am on May 25, 2012 (gmt 0)
Would LinkScrubber inadvertently click on Adsense ads whilst checking links on your site?
3:11 pm on May 25, 2012 (gmt 0)
Since IncredBill is an AdSense publisher himself (and moderator of the AdSense forum here), and created LinkScrubber, I am quite sure AdSense is safe from it.