homepage Welcome to WebmasterWorld Guest from 184.73.40.21
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Newbie - Can spiders go bezerk?
This spider sucks the same page over and over again
Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 281 posted 11:46 am on Jan 12, 2001 (gmt 0)

My mother tonge is French. I am trying my best... :-)

Every five minutes or so, this spider is "Head"ing for the same frame page. I submitted this frame page, wich is essentially a list of URL leading to entry pages, 7 days ago.

198.139.155.8 - - [12/Jan/2001:04:15:17 -0500] "GET /liens/36892731767.html HTTP/1.0" 200 92
198.139.155.8 - - [12/Jan/2001:04:20:54 -0500] "HEAD /liens/36892731767.html HTTP/1.0" 200 29970
198.139.155.8 - - [12/Jan/2001:04:26:32 -0500] "HEAD /liens/36892731767.html HTTP/1.0" 200 29970
198.139.155.8 - - [12/Jan/2001:04:32:25 -0500] "HEAD /liens/36892731767.html HTTP/1.0" 200 29970

This weird behavior started 36 hrs ago and is still going on.

Can someone tell me wich robot it is, and the purpose of thoses visits.

 

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 281 posted 1:11 am on Jan 13, 2001 (gmt 0)

Well, I can't tell you much. The IP is part of a Verio block - [home.verio.com...] .
What is it's user agent?

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 281 posted 2:04 am on Jan 13, 2001 (gmt 0)

I wrote that I was a newbie, now is time for me to prove it!

Pardon my ignorance but I do not know how to get user agent from robots, for this site.

The access to the log of my client's ISP does not display user agent. It is HTML output from a PERL script. They will not allow users to get real log files, use Telnet or customise AnaLog.

I use LogIT PRO v2.2.4 to provide reports for this site. It shows domain names and user agents for human visitors. It does not display UA for robots.

I understand you could use this information but I simply cannot provide it.

As for the bizzare robot it did finally "swallow" at once, all the pages linked from the frame page and keeps on HEAD ing and GET ting it every five minutes.

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 281 posted 2:29 am on Jan 13, 2001 (gmt 0)

>They will not allow users to get real log files, use Telnet or customise AnaLog.

You should stop using that hosting company and find a company that is less restrictive. Those types of restrictions are much too limiting and are keeping you from information that will help you tremendously. By the way, your English is much better than my French.

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 281 posted 3:00 am on Jan 13, 2001 (gmt 0)

Thank you for your efforts.

I feel frustrated by this situation. I am promoting an existing site for a client who has already chosen his host. I did recommend this client another host, but it is all I could do.

I really like this forum. I learn a lot here.

P.S. French is harder to learn than C++ when it is not your mother tonge,
here is a liric line: Mettre en ligne un site Web sans le promouvoir, équivaut à jeter une bouteille à la mer.

Thank's again

rencke

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 281 posted 11:23 am on Jan 13, 2001 (gmt 0)

>Mettre en ligne un site Web sans le promouvoir, équivaut à jeter une bouteille à la mer.

I almost didn't make it through high school 37 years ago because of my poor French. Now let's see if I can get that line right:

"To put a website online without promotion, is like throwing a bottle into the sea." Right?

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 281 posted 12:56 pm on Jan 13, 2001 (gmt 0)

Bravo! 100% right!

I believe your French teacher was tough then. As for my English teacher she was very sexy so I didn't miss a course. :-)

Does it make sense in English? Do we have to specify about the message in the bottle?

The weird robot is gone now. It has been HEAD ing the same file every five minutes for almost tree days. Do someone have a clue on such a behavior?

The mothly report shows it in unresolved numerical addresses, no UA. It has been hitting the same page 6735 times.

grnidone



 
Msg#: 281 posted 12:07 am on Jan 14, 2001 (gmt 0)

>Does it make sense in English?

Boy Howdy! (That is 'Texan' for YES, I agree completely.)

That should go on the quote of the week in Foo.

-G

Just so you know, you speak English better than I do. And that is the only language I know how to speak! ;)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved