Welcome to WebmasterWorld Guest from 54.196.2.131

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

damn404 broken link checker

     
7:03 pm on Aug 30, 2017 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10116
votes: 550


UA: Mozilla/5.0 (compatible; damn404 broken link checker - http://damn404.com/crawlinfo/ )
Protocol: HTTP/1.1
Robots.txt: Yes
Host: hetzner.de
78.46.18.64 - 78.46.18.127
78.46.0.0/15
144.76.0.0 - 144.76.255.255
144.76.0.0/16

Caught this agent scraping files not needed for link validity, so either it is being abused or spoofed.

Previous discussion: [webmasterworld.com...]
8:23 pm on Aug 30, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:14259
votes: 552


So the real thing (previous thread) doesn't ask for robots.txt but the spoofer does? Ha.
8:29 pm on Aug 30, 2017 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10116
votes: 550


Not saying either is the "real thing" really. Without a deeper investigation, any of these UA reports are just speculation they are who they say they are.

The first sighting could have had the robots.txt cached from an earlier visit, or the setting could have been changed. Lots of possibilities, including me missing it.
5:15 pm on Nov 11, 2017 (gmt 0)

New User

joined:Nov 11, 2017
posts:1
votes: 0


Hi keyplyr,

damn404 is crawling and parsing HTML documents to find broken links.
Other media like PDF files are checked with a "HEAD" request if they are linked.

Robots.txt file is generally obeyed, if you can show me a specific file that you think was crawled maliciously give me a shout and I will look into it.
7:27 pm on Nov 11, 2017 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10116
votes: 550


Hi damn404admin and welcome to WebmasterWorld [webmasterworld.com]
12:49 am on Nov 12, 2017 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11080
votes: 106


welcome to WebmasterWorld, damn404admin!