A spider with a referring URL

I knew I'd seen this before

[volatilegx]

I got hit by a Lycos spider which used a referring URL.

The spider was at 202.232.118.51, using the Agent "Lycos_Spider_(T-Rex)" and the referring URL was from [lycos.co.jp...]

I had posted that I had seen a spider with a referring URL before and was asked for evidence... I couldn't find that thread so I'm starting a new one.

Anybody else seeing this happen?

[roscoepico]

Could it be that it was a user using a Lycos user agent who happened to do a search?

[Bolotomus]

I've seen that before too. Frankly, I think spiders should *always* have a referring url. In other words, "how did you find out about this link?" There are some cases where I made a page and kept it to myself, only to later find it in the engines. I would love to learn how they first found out.

I've also seen engines (although not Lycos) which did searches on other search engines, and then spidered the results, and would politely show which searches they used. Usually the search was for some term, from the 'advanced search page,' as it appeared in the last (N) days.

Bolotomus

[volatilegx]

roscoepico,

Nope that IP address is definitely from Lycos.

[Josk]

Yep...its Japanase. Say konichiwa!

Network Information:
a. [Network Number] 202.232.118.0-202.232.119.0
b. [Network Name] LYCOS-JAPAN

[Josk]

I was wondering about this yesterday but my connection to APNIC seemed to be down... I wonder if this will spider english content well?.

[Gorufu]

> I got hit by a Lycos spider which used a referring URL.

Hi volatilegx,

Did the referring URL appear before the UA, Lycos_Spider_(T-Rex), and were they both from the same IP? It is possible that someone at Lycos Japan was searching and found your site then indexed it.

I have several Japanese pages indexed with Lycos Japan and the referring URL's are basically the same. I split them to avoid horizontal scrolling.

144.137.135.209 - - [26/Jul/2001:19:11:50 +1000] "GET /golf/index.html HTTP/1.0" 200 3591
"http://www.lycos.co.jp/cgi-bin/pursuit?query=gold+coast+australia
&cat=jp&encoding=shift-jis" "Mozilla/4.72 [ja] (Win95; I)"

144.137.135.209 - - [26/Jul/2001:19:14:28 +1000] "GET /jpgolf/index.html HTTP/1.0" 200 3827
"http://www.lycos.co.jp/cgi-bin/pursuit?query=
%83I%81%5B%83X%83g%83%89%83%8A%83A+%83S%83%8B%83t
&cat=jp&encoding=shift-jis" "Mozilla/4.72 [ja] (Win95; I)"

The first URL is english keywords and the second is Japanese keywords that are parsed as hex code. The IP address is an Aussie DSL IP that I am currently connected to.

> I wonder if this will spider english content well?.

Hi Josk,

Lycos Japan does index English pages. I don't if they index English sites or just spider all links for sites that have Japanese content.

[volatilegx]

Hi Gorofu,

I don't understand your question..

What I was referring to was a visit by a Lycos spider, not a human. The spider had the Lycos UA "Lycos_Spider_(T-Rex)". What is unusual is that the http header generated by the spider contained a referring URL:

[lycos.co.jp...]