User-Agent: MyLycos

Please, this is all over my stuff

[han solo]

The ip address for this one is 64.89.33.108. It won't resolve, it doesn't look up, and in arin.net it says the b class is owned by Exodus. I looked and looked, and can't dig up any dirt on this one.

Given the number of engines coming out of exodus, I don't really want to filter it without very, very good reason. Littleman, Brett_Tabke, somebody, ANYBODY...I'm kind of freaked out on this one.

Appreciate all the help, and thanks in advance.

(merry christmas :) if I don't say it later)

Cheers,
Han Solo

[msgraph]

Maybe this should help Han

Do you have any news related items on your site?

my.lycos.com offers a personalized news service. What this does is the users, which would come from Lycos, select topics that would interest them. Then every day these robots go out from the user's PC (I think) and gather all of the news that the user selected and returns the info back to them.

I'm not sure if this is what is hitting your site but it might be.

[msgraph]

This might explain it better.

[cyberalert.com...]

Check the section:

Electronic News Services

[han solo]

Hi-thanks for the link. It is interesting, and I've seen their service before. The ip doesn't match their domain, do you have any info that says they subcontract their stuff from exodus?

As for the lycos page, what about that User agent says it's from lycos? They use T-rex, which spiders me regularly, and their c class in arin.net isn't from Exodus, but says Lycos on it clearly.

Sorry, I think you missed. Anyone else have a promising lead? I appreciate your help, though.

Cheers,
Han Solo

[msgraph]

It must be coming from my.lycos.com. Do a tracert on my.lycos.com and you will see that the IP address right before the final destination is:

64.89.36.134

I mean it's way off the path but both the IP above and the IP you are asking about both come out of Santa Clara, CA.

[han solo]

This is why I don't believe it is lycos:

Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)

That is the original user agent in my logs, from the 7th of this month. On the 18th was the first day the user agent changed to the weird lycos thing.

Has anybody else had any experience with this, or have you Msgraph had this one spider your site?

Cheers,
han solo

[littleman]

I can't find anything significant on that IP or UA. Has anyone downloaded the lycos browser? Could it be the UA for that?

[DavidP]

A reverse whois on Exodus shows that 64.89.33.108 is from a Lycos Inc allocated IP block.

[littleman]

It does? I've NSlookuped it, ReverceDNS 'ed it, arin lookedup'ed it, class C'ed it, tracerouted it and didn't get passed Exodus.

Can you show me what you did?

[DavidP]

I used this [rwhois.net] tool setting HostName to rwhois.exodus.net with port 4321. IP number as the query.

I've noticed it be a little cranky sometimes, which I think is the server being queried not the tool.

[msgraph]

[han solo]

Even using that tool, I don't see how it said that was a lycos allocated block.

Second, I am highly suspicious even if it is lycos: they have never, to my knowledge, utilized a user agent such as the one that I found. And why, if it is lycos, would they have been spidering before with a different User agent that sounds like some windows product?

In the race to prove whether or not it is a lycos allocated IP, why doesn't somebody put forth their opinion on this?

Has anyone seen a spider from a legitimate engine running WinHttpRequest.5, which sounds like a windows http request utility for a program, and not a bot. Or better yet, change their user agent mid stream, to fool somebody like myself, or anyone else here who cloaks?

I've seen the mozilla ones, working for all sorts of engines, but they always had an ip that proved the case. Even when Ink was running the NetBSD spider, it still had an Inktomi IP on it.

I do appreciate all of the help. And it has been interesting, to see how quick some have written it off as, "it has to be lycos, the ua says so," even when they always use T-Rex on my stuff. :)

Cheers,
Han Solo

[DavidP]

The tool seems to be unable to connect to Exodus at the moment. You can see the Lycos allocation when its working properly. Important usage with the tool is that you don't allow it to use the default server as this just copies the standard whois you get with NIC - you must specify 'rwhois.exodus.net' to get the additional info.

As you point out, just because it comes from a Lycos IP doesn't identify it as a bot. The WinHTTP user agent (I think) is from a Microsoft server-side component, which would make a bot a possible candidate.

What sort of pattern do you see in the logs? Does it look as if it may be crawling?

[msgraph]

>>The WinHTTP user agent (I think) is from a Microsoft server-side component, which would make a bot a possible candidate.

Isn't it used to execute XML applications?

[littleman]

This thing has hit several of my sites today and yesterday. It looks like it is a machine, not a human. It only hit the root, '/,' of the domains, and didn't call the robots.txt. It is always coming from that same IP. It's a bot of some sort, but I know it hasn't hit any of the pages I've submitted to Lycos.

[Brett_Tabke]

I've gotten hit by this thing a few times in the last month. It has fixated on one site. Stock ie agent name, and pulls about 10 pages a day.