Whoa - Xenu Link Slueth and Yahoo

[Brett_Tabke]

ACK! What's this?

surf-city3.corp.yahoo.com - - [03/Aug/2002:12:31:58 -0500] "GET / HTTP/1.1" 403 289 "-" "Xenu Link Sleuth 1.2b"

What the? Yahoo running Xenu!?

Ah - Yikes - I've got Xenu banned in my htaccess!

[2_much]

Brett, did this happen again? Wow...I can't believe this.

[JamesR]

That would be an easy, free way to check for broken links on a site before inclusion.

[Brett_Tabke]

Definately a yahoo editor running Xenu...

[rogerd]

I think Xenu is popular enough to make one think twice about banning it. I use it to check for link integrity on the sites I manage. I usually manually verify any links reported as being broken, but if I was weeding out a big directory, I might just delete any links that Xenu couldn't load. (Well, maybe not WebmasterWorld's... :)) It never occurred to me that a Yahoo editor would be using it, but people use what they know...

The downside of failing to ban Xenu is its ability to map out your entire site for any clown that runs it, as well as the fact that it can place multi-threaded demands on your server. I dunno...