homepage Welcome to WebmasterWorld Guest from 54.146.175.204
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Hit Hard 876 reqs in 42 min
ip176.danville3.va.pub-ip.psi.net
Son_House

10+ Year Member



 
Msg#: 713 posted 1:03 pm on May 28, 2001 (gmt 0)

This really ticked me off when I saw it. :o Luckily the power went off for an hour so I could chill.

Anyway, I think this might be some one running their own personal bot but I'm not sure. They entered the site by following a link from an se. Then they made 876 requests, 843 of which were for pages in 42 minutes. They did not request robots.txt The site itself only has about 120 pages so they requested the same files many, many times.

If anyone has any info on this I'd really appreciate it. :)

ip176.danville3.va.pub-ip.psi.net
(38.38.33.176)
UA:Mozilla/4.76 [en]C-gatewaynet (Win95; U)
But for most of the requests, they left the ua blank.

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 713 posted 8:13 am on May 29, 2001 (gmt 0)

>pub-ip.psi.net

That an isp customer. Looks like joe user is just spidering you for data.

Son_House

10+ Year Member



 
Msg#: 713 posted 2:18 am on May 30, 2001 (gmt 0)

Thanks for the info Brett :)

I hope joe user slows down his bot and teaches it to know when it already downloaded a page before he comes back again.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved