| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
| 11:08 pm on Mar 30, 2001 (gmt 0)|
This one is home grown.
Host/IP: wfp2.almaden.ibm.com (22.214.171.124)
Pulled 112 pages in a reasonable length of time. Seems to obey the RES.
This one has perked my curiosity.
UA: unlostBot unlostBot@unlost.com.
Anybody ever use the Webclipping robot? I know what it is for but wondered if anybody used their service or had info on them. I've been getting hit from them every day for months.
| 9:32 pm on Apr 3, 2001 (gmt 0)|
That unlost.com front page does grab your attention.
| 9:55 pm on Apr 3, 2001 (gmt 0)|
I had been hit by webclipping.com as well for weeks (every day, deep spidering)
I wrote them numerous emails to ask if they could spider once a week and if they obey robots.txt.
No response. I ended just banning them in my .htaccess.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
© Webmaster World 1996-2014 all rights reserved