homepage Welcome to WebmasterWorld Guest from 54.163.72.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
80.184.32.134 Robot in Germany?
spidered entire site. Very strange
Larryhat




msg:893334
 9:58 am on Oct 19, 2004 (gmt 0)

Can anyone tell me anything about an oddly behaved (apparent) spider using this dns: 80.184.32.134?

It was referred to my site from an or ordinary organic link I have on a similar site. Then it proceeded to browse like a regular visitor .. but ..

It sucked in every last page I have, including the images. I put up one image per page usually. The pattern was totally mechanical.

Index page => menu #1
Menu #1 => page A, then downloads image A.
Back to Index.

Index page => menu #1
Menu #1 => page B, then downloads image B.
Back to Index.

This repeated until all pages on menu #1 were exhausted. Next back to index.html, and the whole silly waltz repeated with menu #2. Then Menus 3 and 4. Finally it went nuts on my sitemap and found all my oddball pages.

This was extremely methodical, with hits coming from 3 to 10 seconds apart. NO human browses like that.

It never looked at robots.txt. There was no referral string or identifier. Spiders usually only suck up my html files. Image bots come later for those.

I've never seen this before, my only clue was the dns.

Any clues? - Larry

 

Hanu




msg:893335
 10:14 am on Oct 19, 2004 (gmt 0)

The IP is registered to FreeNet so it might well be a dialup IP. Did it fetch robot.txt? What was the UserAgent? Maybe it was just IE downloading webpages for offline reading?

Larryhat




msg:893336
 11:30 am on Oct 19, 2004 (gmt 0)

I went thru my entire access_log file. Whatever this is, it did NOT call for robots.txt. I have a robots.txt, which has no restrictions on spidering.

For User agent I only saw "-" meaning unspecified.

It doesn't want to identify itself, and almost tries to mimic a surfer. Only the complete relentless spidering and rate of downloads give it away. - Larry

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved