Welcome to WebmasterWorld Guest from 54.198.118.102

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Freaky spiders

     
11:08 pm on Mar 30, 2001 (gmt 0)

Full Member

joined:Jan 24, 2001
posts:230
votes: 0


This one is home grown.

UA: WebFountain
Host/IP: wfp2.almaden.ibm.com (198.4.83.49)
[swcp.com...]
Pulled 112 pages in a reasonable length of time. Seems to obey the RES.

This one has perked my curiosity.

IP: 212.37.219.147
UA: unlostBot unlostBot@unlost.com.
[unlost.com...]

Anybody ever use the Webclipping robot? I know what it is for but wondered if anybody used their service or had info on them. I've been getting hit from them every day for months.

[webclipping.com...]

9:32 pm on Apr 3, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


That unlost.com front page does grab your attention.
9:55 pm on Apr 3, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 3, 2001
posts:44
votes: 0


I had been hit by webclipping.com as well for weeks (every day, deep spidering)

I wrote them numerous emails to ask if they could spider once a week and if they obey robots.txt.

No response. I ended just banning them in my .htaccess.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members