Welcome to WebmasterWorld Guest from 54.161.92.49

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Xenu Link Sleuth

Should anonymous linkers be blocked?

     
12:26 am on Jul 2, 2012 (gmt 0)

Preferred Member

5+ Year Member

joined:July 22, 2010
posts: 369
votes: 0


Hi,

I understand this is a link checker, possibly looking for 404s. Is it normal for the person doing the checking to hide their identity? What do you make of this?

75.128.105.nn - - [01/Jul/2012:13:21:41 -0400] "HEAD / HTTP/1.1" 301 - "-" "Xenu Link Sleuth/1.3.8"

75.128.105.nn - - [01/Jul/2012:13:22:43 -0400] "HEAD /example HTTP/1.1" 301 - "-" "Xenu Link Sleuth/1.3.8"

75.128.105.nn - - [01/Jul/2012:13:27:13 -0400] "HEAD /example/ HTTP/1.1" 200 - "-" "Xenu Link Sleuth/1.3.8"

Assuming whoever this is has links to my site. It would be nice to know who they are. The IP belongs to Charter.

-- GG
7:40 am on July 6, 2012 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13825
votes: 484


Because speed traps, volume traps and other ways of easily detecting steath UAs don't typically apply to bots that have permission which is why using one of my allowed user agents to gain access would be a breakthrough for the bad bot as validated access disables all other tests and you get a free pass.

... and, conversely, I doubt that more than 1% of robots think that far ahead. Granted, those are the 1% you really have to worry about ;) but you gotta concede that your average robot is too stupid to find its access plate with both hands.
8:22 am on July 6, 2012 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8927
votes: 404


GG - the person running the link scan is not "hiding their identity."

If they visited your site with a browser, the UA string would contain IP, time stamp, request type, referrer, and UA.

That's exactly what is being displayed here, except instead of a browser UA, it is the software UA ("Xenu Link Sleuth/1.3.8") which is what is actually sending the requests. There's no referrer since the software is not coming from another web site it is installed on the owner's machine.
4:01 pm on July 6, 2012 (gmt 0)

Preferred Member

5+ Year Member

joined:July 22, 2010
posts: 369
votes: 0


Blend27, in answer to your question, yes. Hmmm. Curiouser and curiouser. I checked back 6 months in the logs and found that xenu only showed up about once a month, until July. In July it showed up 32 times. So I'm gonna block it. Goodbye Xenu, too much of anything is not good.
4:32 pm on July 6, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5460
votes: 3


July 3,
Stop dinking around and simply add Xenu to your UA deny list.


July 6,
In July it showed up 32 times. So I'm gonna block it. Goodbye Xenu


;)
12:05 am on July 7, 2012 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14663
votes: 99


... and, conversely, I doubt that more than 1% of robots think that far ahead.


Depends on the bot, regular crawlers don't, the underbelly of the internet scrapers do because they'll stop at nothing to get what they want.

Had one scraper that kept getting caught in my traps that actually went to the extreme of reducing his camouflage crawl rate to one page per day so I expanded my time trap to track IPs for 48 hours and sure enough they kept pinging the site around every 24 hours and continued with sequential page requests at that slow pace for hundreds of days.

It was the funniest thing I ever saw.

</thread hijack>
1:36 am on July 7, 2012 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13825
votes: 484


It was the funniest thing I ever saw.

Oh, lord, it's the robotic equivalent of painting the Golden Gate Bridge. By the time they're done they would have to start all over again to pick up the last few years' worth of changes.

Still with us, Grandma? If I remember rightly, you are not in fact anyone's grandma or even grandpa. I suppose it was explained somewhere.
4:38 am on July 9, 2012 (gmt 0)

Preferred Member

5+ Year Member

joined:July 22, 2010
posts: 369
votes: 0


The rumors of my age have been highly exaggerated.
12:47 am on July 24, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1889
votes: 56


It's back from 98.154.143.247(IP in StopForumSpam) - 20 days later...

followed by 72.93.206.97 - same headers.
1:24 am on July 24, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5460
votes: 3


blend,
I've this same IP on July 2nd, perhaps even since.
I've simply been passing over the 403's.

I just chalked this IP and another RR IP as somebody from this forum flexing their muscles, since most every visit seems to coincide with this threads activity ;)

FWIW, we use to have a forum participant that would inject absurd UA's and visit sites because he thought it was cute. Don't recall the identity.
This 39 message thread spans 2 pages: 39
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members