Newbie - Can spiders go bezerk?

This spider sucks the same page over and over again

[Macguru]

My mother tonge is French. I am trying my best... :-)

Every five minutes or so, this spider is "Head"ing for the same frame page. I submitted this frame page, wich is essentially a list of URL leading to entry pages, 7 days ago.

198.139.155.8 - - [12/Jan/2001:04:15:17 -0500] "GET /liens/36892731767.html HTTP/1.0" 200 92
198.139.155.8 - - [12/Jan/2001:04:20:54 -0500] "HEAD /liens/36892731767.html HTTP/1.0" 200 29970
198.139.155.8 - - [12/Jan/2001:04:26:32 -0500] "HEAD /liens/36892731767.html HTTP/1.0" 200 29970
198.139.155.8 - - [12/Jan/2001:04:32:25 -0500] "HEAD /liens/36892731767.html HTTP/1.0" 200 29970

This weird behavior started 36 hrs ago and is still going on.

Can someone tell me wich robot it is, and the purpose of thoses visits.

[littleman]

Well, I can't tell you much. The IP is part of a Verio block - [home.verio.com...] .
What is it's user agent?

[Macguru]

I wrote that I was a newbie, now is time for me to prove it!

Pardon my ignorance but I do not know how to get user agent from robots, for this site.

The access to the log of my client's ISP does not display user agent. It is HTML output from a PERL script. They will not allow users to get real log files, use Telnet or customise AnaLog.

I use LogIT PRO v2.2.4 to provide reports for this site. It shows domain names and user agents for human visitors. It does not display UA for robots.

I understand you could use this information but I simply cannot provide it.

As for the bizzare robot it did finally "swallow" at once, all the pages linked from the frame page and keeps on HEAD ing and GET ting it every five minutes.

[littleman]

>They will not allow users to get real log files, use Telnet or customise AnaLog.

You should stop using that hosting company and find a company that is less restrictive. Those types of restrictions are much too limiting and are keeping you from information that will help you tremendously. By the way, your English is much better than my French.

[Macguru]

Thank you for your efforts.

I feel frustrated by this situation. I am promoting an existing site for a client who has already chosen his host. I did recommend this client another host, but it is all I could do.

I really like this forum. I learn a lot here.

P.S. French is harder to learn than C++ when it is not your mother tonge,
here is a liric line: Mettre en ligne un site Web sans le promouvoir, équivaut à jeter une bouteille à la mer.

Thank's again

[rencke]

>Mettre en ligne un site Web sans le promouvoir, équivaut à jeter une bouteille à la mer.

I almost didn't make it through high school 37 years ago because of my poor French. Now let's see if I can get that line right:

"To put a website online without promotion, is like throwing a bottle into the sea." Right?

[Macguru]

Bravo! 100% right!

I believe your French teacher was tough then. As for my English teacher she was very sexy so I didn't miss a course. :-)

Does it make sense in English? Do we have to specify about the message in the bottle?

The weird robot is gone now. It has been HEAD ing the same file every five minutes for almost tree days. Do someone have a clue on such a behavior?

The mothly report shows it in unresolved numerical addresses, no UA. It has been hitting the same page 6735 times.

[grnidone]

>Does it make sense in English?

Boy Howdy! (That is 'Texan' for YES, I agree completely.)

That should go on the quote of the week in Foo.

-G

Just so you know, you speak English better than I do. And that is the only language I know how to speak! ;)