Luckily the DNS had computer_science division or something. I didn't mind them grabbing the stuff if it was for some research but after time I was getting sick of them eating up bandwidth. I just went to their CS Dept. page on their site and asked the Head Techie if they were doing some sort of research.
He told me it was coming out of a lab classroom and that he would look into it. After he replied it stopped for a few days and then started back up again. It was like clockwork, everyday from 1-3 p.m. So, remembering my old college days, I assumed it was some student using their high-speed connection to grab a bunch of stuff. If it was a researcher I imagine that they would dedicate a computer and let it run day and night. But who knows.
Since it was comming out of a CS I lab, I just went ahead and banned their IP.
>>Last time I got one of their spiders, it was libwww/version#. Now it's this java thing. Makes one wonder if they are really doing something, or just experimenting with different types of technology.
Could be in my case. That's probably how the guys that made Google started out.
I don't think there is anything to worry about unless they are grabbing a lot of pages like a bot and where they are coming from seems suspicious. It is probably some open source page downloading tool/web browser out there that spits out this UA when it's used.
If there are any Java2 SDK developers or Java platform user out there I'm sure they can give us a better insight