homepage Welcome to WebmasterWorld Guest from 50.16.130.188
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Freaky spiders
Froggyman




msg:395754
 11:08 pm on Mar 30, 2001 (gmt 0)

This one is home grown.

UA: WebFountain
Host/IP: wfp2.almaden.ibm.com (198.4.83.49)
[swcp.com...]
Pulled 112 pages in a reasonable length of time. Seems to obey the RES.

This one has perked my curiosity.

IP: 212.37.219.147
UA: unlostBot unlostBot@unlost.com.
[unlost.com...]

Anybody ever use the Webclipping robot? I know what it is for but wondered if anybody used their service or had info on them. I've been getting hit from them every day for months.

[webclipping.com...]

 

littleman




msg:395755
 9:32 pm on Apr 3, 2001 (gmt 0)

That unlost.com front page does grab your attention.

jetsetter




msg:395756
 9:55 pm on Apr 3, 2001 (gmt 0)

I had been hit by webclipping.com as well for weeks (every day, deep spidering)

I wrote them numerous emails to ask if they could spider once a week and if they obey robots.txt.

No response. I ended just banning them in my .htaccess.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved