homepage Welcome to WebmasterWorld Guest from 54.166.113.249
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Freaky spiders
Froggyman



 
Msg#: 515 posted 11:08 pm on Mar 30, 2001 (gmt 0)

This one is home grown.

UA: WebFountain
Host/IP: wfp2.almaden.ibm.com (198.4.83.49)
[swcp.com...]
Pulled 112 pages in a reasonable length of time. Seems to obey the RES.

This one has perked my curiosity.

IP: 212.37.219.147
UA: unlostBot unlostBot@unlost.com.
[unlost.com...]

Anybody ever use the Webclipping robot? I know what it is for but wondered if anybody used their service or had info on them. I've been getting hit from them every day for months.

[webclipping.com...]

 

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 515 posted 9:32 pm on Apr 3, 2001 (gmt 0)

That unlost.com front page does grab your attention.

jetsetter

10+ Year Member



 
Msg#: 515 posted 9:55 pm on Apr 3, 2001 (gmt 0)

I had been hit by webclipping.com as well for weeks (every day, deep spidering)

I wrote them numerous emails to ask if they could spider once a week and if they obey robots.txt.

No response. I ended just banning them in my .htaccess.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved