Forum Moderators: open
Oh great god of WebmasterWorld, please de-deprecate my favorite forum :)
I've been thinking about opening my own forum devoted specifically to the discussion of search engine spiders. I already have one as an adjunct to one of my commercial websites, but I want to make one that is not commercialized at all.
However, if the discussion is not carried on here, there likely won't be the kind of activity that's needed to keep it alive.
66.228.91.*** - - [12/Feb/2004:10:49:56 -0800] "GET /robots.txt HTTP/1.1" 200 1524 "-" "NLese"
66.228.91.*** - - [12/Feb/2004:10:49:57 -0800] "GET / HTTP/1.1" 200 20402 "-" "NLese" Sure, it asks for robots.txt, but from what I see this "NLese" is available for download.
http://216.239.57.104/search?q=cache:vlhserG4Bh0J:special.northernlight.com/downloads/ESE20030712.pdf+&hl=en&ie=UTF-8
If not here, where?
We need the IP Number in order to determine the legitimacy of the bot. While those who operate these critters have every right to protect the name of thier unit, that should also be balanced with an opposing view as to the bot behaving in a manner consistent with our wishes.
I just don't see the future of the Internet being one ruled by spiders and bots, be they malicious or not, nor do I see the webmasters of the World enduring these assaults simply as another 'part of doing business'.