homepage Welcome to WebmasterWorld Guest from 54.221.175.46
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Spiders
He.net, anybody know who he is.
woodrow




msg:404528
 3:28 pm on Apr 26, 2001 (gmt 0)

Hello,

He.net or analysis.he.net has been to several of my sites doing deep crawls. Any one know who it is?

Thanks

 

bobriggs




msg:404529
 4:37 pm on Apr 26, 2001 (gmt 0)

Been seen before, no definitive answers:

[webmasterworld.com...]

littleman




msg:404530
 5:05 pm on Apr 26, 2001 (gmt 0)

They've been making the rounds laity.
[webmasterworld.com...]
This is one of the post that talks about it. I think there are two others. If you plug in their IP it will throw you to a splash page that talks about the server farm in Fremont. I know they are crawling from two separate IPs. At first I thought they might be doing some clever marketing by causing a buz with their crawling. But I really don't know what they are up to yet.

theperlyking




msg:404531
 8:20 pm on Apr 28, 2001 (gmt 0)

Its hammering one of my sites, doing faulty requests for fragments of javascript and generally annoying me :(
Strange thing is it has a UA of Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0).
I am very tempted to ban them.

digitalgirl




msg:404532
 9:39 pm on Apr 29, 2001 (gmt 0)

Any ideas if it's worth blocking this IP...is there a good way of banning them *as if there is a good way of banning, but hey* Thanks for all the help ;)

theperlyking




msg:404533
 9:48 pm on Apr 29, 2001 (gmt 0)

If you are using a *nix system you can use the .htaccess file to ban them and it will work unless they start coming from a different IP.

bot_watcher




msg:404534
 11:13 pm on Apr 29, 2001 (gmt 0)

I contacted the huuricane electric abuse department. After they deep crawled my site super fast 3 times. Here was there response: Thank you for your feedback/interest. analysis.he.net is an experimental
web indexing tool. We take a number of precautions to ensure that the
load we cause your web server is minimized such as only accessing one URL
at your domain name at a time. We appologize for any confusion we may
have caused.
And here is the email address of the guy who responded: Scott Nelson <scottn@he.net>

Froggyman




msg:404535
 1:33 am on Apr 30, 2001 (gmt 0)

The email was a big [url=www.he.net/~scottn/]help[/url]. Finally something to work with...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved