homepage Welcome to WebmasterWorld Guest from 54.205.247.203
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Hit Hard 876 reqs in 42 min
ip176.danville3.va.pub-ip.psi.net
Son_House




msg:408783
 1:03 pm on May 28, 2001 (gmt 0)

This really ticked me off when I saw it. :o Luckily the power went off for an hour so I could chill.

Anyway, I think this might be some one running their own personal bot but I'm not sure. They entered the site by following a link from an se. Then they made 876 requests, 843 of which were for pages in 42 minutes. They did not request robots.txt The site itself only has about 120 pages so they requested the same files many, many times.

If anyone has any info on this I'd really appreciate it. :)

ip176.danville3.va.pub-ip.psi.net
(38.38.33.176)
UA:Mozilla/4.76 [en]C-gatewaynet (Win95; U)
But for most of the requests, they left the ua blank.

 

Brett_Tabke




msg:408784
 8:13 am on May 29, 2001 (gmt 0)

>pub-ip.psi.net

That an isp customer. Looks like joe user is just spidering you for data.

Son_House




msg:408785
 2:18 am on May 30, 2001 (gmt 0)

Thanks for the info Brett :)

I hope joe user slows down his bot and teaches it to know when it already downloaded a page before he comes back again.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved